What happened in the Kafka community in February 2019?
Releases:
– 2.2.0: On February 23, 2019, Matthias J. Sax cut the first Release Candidate but the community found a blocker JIRA that required a new RC. RC1 was cut on March 1, and the voting process is currently opened. As a result, this new minor release should be available early March.
– 2.1.1: On February 9, 2019, Colin McCabe released this bugfix version. It contains dozens of fixes including a 4 blocker JIRAs.
KIPs:
Last month, the community submitted 13 KIPs (KIP-425 to KIP-437), and these are the ones that caught my eye:
KIP-425: Add some Log4J Kafka Appender Properties for Producing to Secured Brokers
Sending application logs to Kafka is a very common use case, and, in fact, Kafka includes a Log4J appender. However, currently, it does not support producing to brokers that use SASL, so this KIP proposes adding a couple of new properties to enable users to configure the appender to use SASL.
KIP-429: Smooth Auto-Scaling for Kafka Streams
In the past few months, the community started looking at ways to improve the rebalance mechanism used by clients to distribute work. One potential option is to use incremental cooperative rebalances. This KIP aims at bringing such a mechanism to Kafka Streams in order to support easy scaling without downtime.
KIP-430: Return Authorized Operations in Describe Responses
At the moment, users have no way to determine which operations they can do on each Kafka resource. Basically, the only way to figure it out is to try the operation and see that it succeeds or if the response contains an authorization error. The goal of this KIP is to include authorization details in the Metadata and DescribeGroups responses so allowed operations are retrievable through the API.
KIP-435: Incremental Partition Reassignment
Currently, when a replicated partition is reassigned, all new replicas will have to be in-sync before any of the old replicas are deleted. As a result, this can cause significant load when reassigning a partition with three replicas to three different brokers as, at worst, Kafka will have to maintain six replicas. This KIP aims at optimizing the reassignment logic to decrease the impact of large reassignments and to delete replicas incrementally as new replicas come online.
Blogs:
-
https://blog.gojekengineering.com/how-kafka-solved-a-culture-problem-at-gojek-600e690e8eb1
-
https://medium.com/bakdata/fluent-kafka-streams-tests-e641785171ec
IBM Event Streams for Cloud is Apache Kafka-as-a-service for IBM Cloud.