Open Source

Kafka Monthly Digest: February 2019

Share this post:

What happened in the Kafka community in February 2019?

Releases:

– 2.2.0: On February 23, 2019, Matthias J. Sax cut the first Release Candidate but the community found a blocker JIRA that required a new RC. RC1 was cut on March 1, and the voting process is currently opened. As a result, this new minor release should be available early March.

– 2.1.1: On February 9, 2019, Colin McCabe released this bugfix version. It contains dozens of fixes including a 4 blocker JIRAs. See the release notes for the full list of changes.

KIPs:

Last month, the community submitted 13 KIPs (KIP-425 to KIP-437), and these are the ones that caught my eye:

KIP-425: Add some Log4J Kafka Appender Properties for Producing to Secured Brokers
Sending application logs to Kafka is a very common use case, and, in fact, Kafka includes a Log4J appender. However, currently, it does not support producing to brokers that use SASL, so this KIP proposes adding a couple of new properties to enable users to configure the appender to use SASL.

KIP-429: Smooth Auto-Scaling for Kafka Streams
In the past few months, the community started looking at ways to improve the rebalance mechanism used by clients to distribute work. One potential option is to use incremental cooperative rebalances. This KIP aims at bringing such a mechanism to Kafka Streams in order to support easy scaling without downtime.

KIP-430 – Return Authorized Operations in Describe Responses
At the moment, users have no way to determine which operations they can do on each Kafka resource. Basically, the only way to figure it out is to try the operation and see that it succeeds or if the response contains an authorization error. The goal of this KIP is to include authorization details in the Metadata and DescribeGroups responses so allowed operations are retrievable through the API.

KIP-435: Incremental Partition Reassignment
Currently, when a replicated partition is reassigned, all new replicas will have to be in-sync before any of the old replicas are deleted. As a result, this can cause significant load when reassigning a partition with three replicas to three different brokers as, at worst, Kafka will have to maintain six replicas. This KIP aims at optimizing the reassignment logic to decrease the impact of large reassignments and to delete replicas incrementally as new replicas come online.

Blogs:

 


IBM Event Streams for Cloud is Apache Kafka-as-a-service for IBM Cloud.

Get started with IBM Event Streams

Software Engineer

More Open Source stories
March 21, 2019

Introducing IBM MQ v9.1.2

The IBM MQ team has been working hard on a number of exciting enhancements and new capabilities to bring you the second Continuous Delivery release of MQ v9.1.

Continue reading

March 12, 2019

Expanding Data Warehouse Capabilities for the IBM Hybrid Data Management Platform

The IBM Hybrid Data Management Platform is expanding capabilities with both the Flex and Hybrid Flex plans. These two types of warehousing solutions will help you optimize your hybrid cloud architectures in terms of both performance and cost-savings

Continue reading

March 11, 2019

Worker Node Auto-Scaling GA in IBM Cloud Kubernetes Service

We're extremely excited to announce the general availability of worker node auto-scaling in IBM Cloud Kubernetes Service.

Continue reading