Open Source

Kafka Monthly Digest: December and 2018 in Review

Share this post:

What happened in the Kafka community in December 2018?

In this post, I will cover what happened in the Apache Kafka community in December as well as some of the milestones the project and community reached in 2018!

Kafka Releases:

No new releases this month.

KIPs:

Last month, the community submitted 13 KIPs (KIP-399 to KIP-411). These are the ones that caught my eye:

KIP-402: Improve fairness in SocketServer processors
Currently, SocketServer favors handling new connections over existing ones. Occasionally, like during a sudden connection storm, this can affect already active clients as the processing of their requests is delayed. This KIP proposes improving this behaviour by making Processors only handle a limited number of new connections before processing existing connections.

KIP-407: Kafka Connect support override worker kafka api configuration with connector configuration that post by rest api
The goal of this KIP is to allow overriding some of the Connect runtime configurations in connectors configuration. The main use case is overriding client (producers for source connectors and consumers for sink connectors) configurations used by the Connect runtime.

KIP-409: Allow creating under-replicated topics and partitions
At the moment, the controller will only create a topic if it is able to create all replicas immediately. This is a pessimistic approach, and, in a few scenarios, it would be valuable to allow deferring the creation of some replicas and still create the topic. For example, in many cases, at least “min.in.sync” replicas are enough. Also, this would allow topic creation when some brokers are unavailable (during a rolling restart) in small clusters.

Blogs:

 

Milestones that the project and community achieved in 2018

Releases in 2018:

The project followed its time-based release plan. Consequently, it released three major versions: 1.1.0, 2.0.0 and 2.1.0.

In addition, four bugfixes were also released: 0.10.2.2, 0.11.0.3, 1.1.1 and 2.0.1.

Kafka Releases in 2018

KIPs in 2018:

Since the KIP process was introduced in 2015, it has raised in popularity each year. In the past 12 months, the community raised over 160 KIPs, almost one ever other day, which was an all-time high!

KIPs each years

Code and contributors

In the past 12 months, over 200 unique contributors made more than 1200 commits.

Lines of Code per version

Committers and PMC:

In 2018, four very active contributors joined the Committers team:

  • Manikumar Reddy
  • Colin McCabe
  • Dong Lin
  • Matthias J. Sax

Likewise, two Committers also joined the Apache Kafka PMC:

  • Rajini Sivaram
  • Dong Lin

IBM Event Streams for Cloud is Apache Kafka-as-a-service for IBM Cloud.

Get started with IBM Event Streams

Software Engineer

More Open Source stories
April 18, 2019

Load Balancing API Calls Across Regions with IBM Cloud Internet Services and Cloud API Gateway

In this article, we'll explore load balancing traffic across two geographically-separated backends built on IBM Cloud Functions. We'll use the IBM Cloud API Gateway to deploy the same API definition in both regions, and then intelligently route traffic with IBM Cloud Internet Services.

Continue reading

April 10, 2019

Improving Your Event Streaming Experience with IBM Event Streams v2019.1.1

We're excited to announce the availability of IBM Event Streams v2019.1.1, another step in our goal to provide clients with the most efficient way to deploy and run Apache Kafka (while leveraging their existing IT systems) so that so they can ultimately build more differentiated, intelligent, event-driven experiences for their customers.

Continue reading

March 21, 2019

Introducing IBM MQ v9.1.2

The IBM MQ team has been working hard on a number of exciting enhancements and new capabilities to bring you the second Continuous Delivery release of MQ v9.1.

Continue reading