Open Source

Kafka Monthly Digest: December and 2018 in Review

Share this post:

What happened in the Kafka community in December 2018?

In this post, I will cover what happened in the Apache Kafka community in December as well as some of the milestones the project and community reached in 2018!

Kafka Releases:

No new releases this month.

KIPs:

Last month, the community submitted 13 KIPs (KIP-399 to KIP-411). These are the ones that caught my eye:

KIP-402: Improve fairness in SocketServer processors
Currently, SocketServer favors handling new connections over existing ones. Occasionally, like during a sudden connection storm, this can affect already active clients as the processing of their requests is delayed. This KIP proposes improving this behaviour by making Processors only handle a limited number of new connections before processing existing connections.

KIP-407: Kafka Connect support override worker kafka api configuration with connector configuration that post by rest api
The goal of this KIP is to allow overriding some of the Connect runtime configurations in connectors configuration. The main use case is overriding client (producers for source connectors and consumers for sink connectors) configurations used by the Connect runtime.

KIP-409: Allow creating under-replicated topics and partitions
At the moment, the controller will only create a topic if it is able to create all replicas immediately. This is a pessimistic approach, and, in a few scenarios, it would be valuable to allow deferring the creation of some replicas and still create the topic. For example, in many cases, at least “min.in.sync” replicas are enough. Also, this would allow topic creation when some brokers are unavailable (during a rolling restart) in small clusters.

Blogs:

 

Milestones that the project and community achieved in 2018

Releases in 2018:

The project followed its time-based release plan. Consequently, it released three major versions: 1.1.0, 2.0.0 and 2.1.0.

In addition, four bugfixes were also released: 0.10.2.2, 0.11.0.3, 1.1.1 and 2.0.1.

Kafka Releases in 2018

KIPs in 2018:

Since the KIP process was introduced in 2015, it has raised in popularity each year. In the past 12 months, the community raised over 160 KIPs, almost one ever other day, which was an all-time high!

KIPs each years

Code and contributors

In the past 12 months, over 200 unique contributors made more than 1200 commits.

Lines of Code per version

Committers and PMC:

In 2018, four very active contributors joined the Committers team:

  • Manikumar Reddy
  • Colin McCabe
  • Dong Lin
  • Matthias J. Sax

Likewise, two Committers also joined the Apache Kafka PMC:

  • Rajini Sivaram
  • Dong Lin

IBM Event Streams for Cloud is Apache Kafka-as-a-service for IBM Cloud.

Get started with IBM Event Streams

Software Engineer

More Open Source stories
March 21, 2019

Introducing IBM MQ v9.1.2

The IBM MQ team has been working hard on a number of exciting enhancements and new capabilities to bring you the second Continuous Delivery release of MQ v9.1.

Continue reading

March 8, 2019

Kafka Monthly Digest: February 2019

Here's what happened in the Kafka community in February 2019.

Continue reading

February 8, 2019

Authenticating Users with Cloud Functions, API Gateway, and App ID

Authentication and authorization are key components for any web app, and APIs built as cloud functions are no exception. This article will show you how to build an IBM Cloud Function API and then use IBM Cloud App ID to protect the function with the IBM Cloud API Gateway—no code change required.

Continue reading