By Mickael Maison 2 min read
July 3, 2019

In this post, I will cover what happened in the Kafka Community in June 2019.

Releases

After three Release Candidates, Colin McCabe released Apache Kafka 2.3.0 on June 25. This new minor version brings a number of interesting features:

Broker:

  • The new IncrementalAlterConfigs API allows you to update only the desired configurations. The AlterConfigs API, which requires you to always specify all configurations, is now deprecated. (KIP-339)
  • The fairness of network processors has been improved and now they prioritize existing connections over incoming ones. This helps scenarios where a massive amount of new connections would significantly impact brokers availability. (KIP-402)
  • Brokers are now able to start faster due to log loading optimizations. (KAFKA-7283)
  • The replication protocol has been hardened. (KIP-461)
  • The monitoring of under-replicated partitions has been improved via new metrics and a new flag for the kafka-topics tool. (KIP-351, KIP-427)

Clients/Connect/Streams:

  • Clients can now easily find which operations they are authorized to perform via the describe methods of the Admin API. (KIP-430)
  • All clients can now use external configurations that are resolved automatically at runtime. (KIP-421)
  • The rebalancing mechanism of Kafka Connect has been improved to be incremental and cooperative. This will reduce the amount and duration of rebalances. (KIP415)
  • Kafka Streams now has built-in in-memory window and session stores. (KIP428, KIP445)

As always, the full release notes are available on apache.org and the release plan is on the wiki.

KIPs

Last month, the community submitted nine KIPs (KIP-477 to KIP-485), and these are the ones that caught my eye:

  • KIP-480: Sticky Partitioner: At the moment, when producing records without keys, Producers assign partitions in a round-robin fashion. While this spreads records evenly across partitions, in many cases, it’s not using the network optimally since batches are often sent before being full. The goal of this KIP is to favour filling up batches by only changing the target partition after each batch, instead of after each record.
  • KIP-482: The Kafka Protocol should Support Optional Fields: The current Kafka protocol specification defines fixed payloads for all the protocol messages. Every time a field is added in a message, a new message version is defined. However, even if a field is not set, it needs to be included and takes some bytes. This KIP’s goal is to make the protocol more flexible and efficient by supporting optional fields.
  • KIP-484: Expose metrics for group and transaction metadata loading duration: When the coordinator for a group changes (due to a broker restarting, for example), the new coordinator has to load the group metadata from the __consumer_offsets partitions. This can take some time, during which groups are idle. This KIP proposes adding metrics to track the duration of such reloads so it’s easy to identify when this happens and how long it takes.

Blogs

IBM Event Streams for Cloud is Apache Kafka-as-a-service for IBM Cloud.

Get started with IBM Event Streams

Categories

Announcements  |  integration  |  Open source  |  open-source  | 
Mickael Maison
Software Engineer

More from Announcements
September 21, 2023

IBM TechXchange underscores the importance of AI skilling and partner innovation

3 min read - Generative AI and large language models are poised to impact how we all access and use information. But as organizations race to adopt these new technologies for business, it requires a global ecosystem of partners with industry expertise to identify the right enterprise use-cases for AI and the technical skills to implement the technology. During TechXchange, IBM's premier technical learning event in Las Vegas last week, IBM Partner Plus members including our Strategic Partners, resellers, software vendors, distributors and service…

August 29, 2023

Introducing Inspiring Voices, a podcast exploring the impactful journeys of great leaders

< 1 min read - Learning about other people's careers, life challenges, and successes is a true source of inspiration that can impact our own ambitions as well as life and business choices in great ways. Brought to you by the Executive Search and Integration team at IBM, the Inspiring Voices podcast will showcase great leaders, taking you inside their personal stories about life, career choices and how to make an impact. In this first episode, host David Jones, Executive Search Lead at IBM, brings…

August 18, 2023

IBM watsonx Assistant and NICE CXone combine capabilities for a new chapter in CCaaS

5 min read - In an age of instant everything, ensuring a positive customer experience has become a top priority for enterprises. When one third of customers (32%) say they will walk away from a brand they love after just one bad experience (source: PWC), organizations are now applying massive investments to this experience, particularly with their live agents and contact centers.  For many enterprises, that investment includes modernizing their call centers by moving to cloud-based Contact Center as a Service (CCaaS) platforms. CCaaS solutions…

August 2, 2023

See what’s new in SingleStoreDB with IBM 8.0

3 min read - Despite decades of progress in database systems, builders have compromised on at least one of the following: speed, reliability, or ease. They have two options: one, they could get a document database that is fast and easy, but can’t be relied on for mission-critical transactional applications. Or two, they could rely on a cloud data warehouse that is easy to set up, but only allows lagging analytics. Even then, each solution lacks something, forcing builders to deploy other databases for…