In this 18th edition of the Kafka Monthly Digest, I will cover what happened in the Kafka Community in July.
For details about Kafka 2.3.0 that was released in June, see last month’s Community Digest.
Releases
No new releases this month.
KIPs
Last month, the community submitted 14 KIPs (KIP-486 to KIP-499) and these are the ones that caught my eye:
KIP-487: Client-side Automatic Topic Creation on Producer: Historically, Kafka brokers have allowed auto topic creations. Now it is recommended to not rely on this feature and, instead, use the AdminClient API to create topics. This KIP aims to move the auto-creation further away from brokers and provide it in the Producer. If enabled, the Producer would automatically create topics that don’t exist.
KIP-488: Clean up Sum,Count,Total Metrics: Metrics are a critical part of monitoring Kafka but it’s key to be able to reason about them and clearly identify what they track. Unfortunately, at the moment, it’s a bit of a mess, as we have Counts, Totals, Sums, SampledTotals, and CumulativeSums, and the relations between them are complex. The goal of this KIP is to expose four clearly named types to remove ambiguity.
KIP-496: Administrative API to delete consumer offsets: KIP-211 was introduced in Kafka 2.1 to prevent consumer offsets from being deleted while a group is active. In some cases, it could be useful to be able to delete consumer offsets even if the group is still active. For example, if the subscription changed, offsets that are not meaningful anymore can pollute lag monitoring. This KIP proposes adding a new API “OffsetDelete” and expose it via the AdminClient to handle those use cases.
KIP-497: Add inter-broker API to alter ISR: The leader and in-sync replica (ISR) data is stored in Zookeeper in a ZNode per partition. At the moment, the Controller is responsible for updating the leader data, but it’s the leader that is responsible for updating the ISR data. This requires extra care to ensure updates are correctly protected. This KIP aims at fixing this issue by making the controller do both types of updates. A new API will be added (AlterISR) to allow leaders to send their ISR data to the controller.
Blogs
- How Kafka Enables Real-Time Stream Processing, Part 1
- How Kafka Enables Real-Time Stream Processing – Part 2
- Communicating Data Changes Across Service Boundaries… Safely!
Get started
IBM Event Streams for Cloud is Apache Kafka-as-a-service for IBM Cloud.