By Mickael Maison 2 min read
December 5, 2018

What happened in the Kafka community in November 2018?

Kafka Releases:

  • 2.1.0: Dong Ling released this new version on November 21, 2018, and it contains a number of interesting new features:

    • Java 11 support: Apache Kafka is keeping up with the faster JVM release cycle. See the OpenJDK page for Java 11 for a list of new features.

    • Support for Zstandard compression: Zstd offers great compression ratios and is also really fast. Consequently, on most workloads, it can provide significant throughput improvements compared to Snappy, GZIP, and LZ4. See KIP-110 for more details.

    • Updated replication protocol: The work to strengthen the inter-broker protocol continued in this release. TLA+, a formal specification language, was used to model and verify the replication protocol.

    • A ton of client improvements: This includes better timeouts for Producers, access to AdminClient metrics, new ACL for listing consumer groups, and DNS improvements for Kerberos and Cloud environments.

    • Improved Streams API: This includes a few new metrics, cleanups of StoreSuppliers APIs, and better timestamp synchronization.

  • 2.0.1: This release came out on the November 9, 2018, and contains 51 fixes. Thanks to Manikumar Reddy for running this release.

Finally, you can access the release plan on the wiki for the full details about each release: 2.0.1 and 2.1.0.

KIPs:

Last month, the community submitted 12 KIPs (KIP-387 to KIP-398). These are the ones that caught my eye:

KIP-388: Add observer interface to record request and response
This KIP proposes adding an interface to brokers to observe requests and responses. The goal is to allow access to information that cannot be retrieved using existing metrics like latency per topic or bytes transmitted per principal.

KIP-390: Allow fine-grained configuration for compression
Compression operations involve a tradeoff between speed and compressed size. Therefore compression algorithms have the concept of compression level to enable adjusting the tradeoff. This KIP’s goal is to allow users to specify the compression level they desire when configuring compression in Producers or Brokers instead of always using the default.

KIP-391: Allow producing with offsets for cluster replication
Replicating data between clusters is a very common operation. However, one difficulty to handle is consumer offsets as messages are likely to have different offsets in each cluster. The proposal of this KIP is to support replicating messages with their offsets. This would enable you to easily replicate committed offsets and allow consumers to rely on them even when switching clusters.

KIP-392: Allow consumers to fetch from closest replica
Currently, Producers and Consumers have to connect to partition leaders in order to send or receive messages. This KIP proposes to give some flexibility to Consumers and allow them to fetch messages from other replicas as long as they are in-sync. In addition to potentially spreading client load on more brokers, this is especially important when a Kafka cluster is spanning multiple availability zones because Consumers would be able to fetch from the closest replica.

Blogs:

IBM Event Streams for Cloud is Apache Kafka-as-a-service for IBM Cloud.

Get started with IBM Event Streams

Categories

Announcements  |  integration  |  Open source  |  open-source  | 
Mickael Maison
Software Engineer

More from Announcements
September 21, 2023

IBM TechXchange underscores the importance of AI skilling and partner innovation

3 min read - Generative AI and large language models are poised to impact how we all access and use information. But as organizations race to adopt these new technologies for business, it requires a global ecosystem of partners with industry expertise to identify the right enterprise use-cases for AI and the technical skills to implement the technology. During TechXchange, IBM's premier technical learning event in Las Vegas last week, IBM Partner Plus members including our Strategic Partners, resellers, software vendors, distributors and service…

August 29, 2023

Introducing Inspiring Voices, a podcast exploring the impactful journeys of great leaders

< 1 min read - Learning about other people's careers, life challenges, and successes is a true source of inspiration that can impact our own ambitions as well as life and business choices in great ways. Brought to you by the Executive Search and Integration team at IBM, the Inspiring Voices podcast will showcase great leaders, taking you inside their personal stories about life, career choices and how to make an impact. In this first episode, host David Jones, Executive Search Lead at IBM, brings…

August 18, 2023

IBM watsonx Assistant and NICE CXone combine capabilities for a new chapter in CCaaS

5 min read - In an age of instant everything, ensuring a positive customer experience has become a top priority for enterprises. When one third of customers (32%) say they will walk away from a brand they love after just one bad experience (source: PWC), organizations are now applying massive investments to this experience, particularly with their live agents and contact centers.  For many enterprises, that investment includes modernizing their call centers by moving to cloud-based Contact Center as a Service (CCaaS) platforms. CCaaS solutions…

August 2, 2023

See what’s new in SingleStoreDB with IBM 8.0

3 min read - Despite decades of progress in database systems, builders have compromised on at least one of the following: speed, reliability, or ease. They have two options: one, they could get a document database that is fast and easy, but can’t be relied on for mission-critical transactional applications. Or two, they could rely on a cloud data warehouse that is easy to set up, but only allows lagging analytics. Even then, each solution lacks something, forcing builders to deploy other databases for…