Kafka Monthly Digest: May 2019

5 min read

By: Mickael Maison

What happened in the Kafka community in May 2019?

Releases:

  • 2.3.0: The release process of Kafka 2.3.0 continued. KIP freeze happened on May 13 and feature freeze on May 17. On May 20, Colin McCabe cut the 2.3 branch. The release date is still planned for mid-June.
  • 2.2.1: A vote for RC0 started on May 8, but an issue was found. The vote for RC1 was successful and Vahid Hashemian released 2.2.1 on the June 3.

KIPs:

Last month, the community submitted 12 KIPs (KIP-465 to KIP-476), and these are the ones that caught my eye:

KIP-466: Add support for List<T> serialization and deserialization

It is relatively common to want to send a record containing a list of objects. At the moment, users have to either convert the List into a supported type (ByteBuffer or ByteArray for example) or provide their own SerDes. This KIP proposes adding a Serializer and a Deserializer for the List<T> Java type to simplify this use case.

KIP-467: Augment ProduceResponse error messaging for specific culprit records

Upon receiving records, Kafka brokers perform some validations to ensure all records are valid. At the moment, if a record looks corrupted, brokers reject the whole batch containing that record, making it hard to identify exactly what happened. The goal of this KIP is to easily identify bad records and improve the handling of such errors.

KIP-469: Pre-registration support for static members

After KIP-345, which introduced static membership in Consumer Group, this KIP aims to reduce potential rebalances even further. This KIP suggests a new Admin API to pre-register group members in order to only rebalance once all expected members have joined, instead of when each member joins.

KIP-471: Expose RocksDB metrics in Kafka Streams

RocksDB is the default state store for Streams. It has a number of metrics that can be really useful to find performance bottlenecks and tune it, but at the moment, users have to gather them explicitly. This KIP proposes exposing the most common RocksDB metrics directly with the Streams metrics.

Blogs:

IBM Event Streams for Cloud is Apache Kafka-as-a-service for IBM Cloud.

Get started with IBM Event Streams

Be the first to hear about news, product updates, and innovation from IBM Cloud