Announcements
Open source
July 3, 2019 By Mickael Maison 2 min read

In this post, I will cover what happened in the Kafka Community in June 2019.

Releases

After three Release Candidates, Colin McCabe released Apache Kafka 2.3.0 on June 25. This new minor version brings a number of interesting features:

Broker:

  • The new IncrementalAlterConfigs API allows you to update only the desired configurations. The AlterConfigs API, which requires you to always specify all configurations, is now deprecated. (KIP-339)
  • The fairness of network processors has been improved and now they prioritize existing connections over incoming ones. This helps scenarios where a massive amount of new connections would significantly impact brokers availability. (KIP-402)
  • Brokers are now able to start faster due to log loading optimizations. (KAFKA-7283)
  • The replication protocol has been hardened. (KIP-461)
  • The monitoring of under-replicated partitions has been improved via new metrics and a new flag for the kafka-topics tool. (KIP-351, KIP-427)

Clients/Connect/Streams:

  • Clients can now easily find which operations they are authorized to perform via the describe methods of the Admin API. (KIP-430)
  • All clients can now use external configurations that are resolved automatically at runtime. (KIP-421)
  • The rebalancing mechanism of Kafka Connect has been improved to be incremental and cooperative. This will reduce the amount and duration of rebalances. (KIP415)
  • Kafka Streams now has built-in in-memory window and session stores. (KIP428, KIP445)

As always, the full release notes are available on apache.org and the release plan is on the wiki.

KIPs

Last month, the community submitted nine KIPs (KIP-477 to KIP-485), and these are the ones that caught my eye:

  • KIP-480: Sticky Partitioner: At the moment, when producing records without keys, Producers assign partitions in a round-robin fashion. While this spreads records evenly across partitions, in many cases, it’s not using the network optimally since batches are often sent before being full. The goal of this KIP is to favour filling up batches by only changing the target partition after each batch, instead of after each record.
  • KIP-482: The Kafka Protocol should Support Optional Fields: The current Kafka protocol specification defines fixed payloads for all the protocol messages. Every time a field is added in a message, a new message version is defined. However, even if a field is not set, it needs to be included and takes some bytes. This KIP’s goal is to make the protocol more flexible and efficient by supporting optional fields.
  • KIP-484: Expose metrics for group and transaction metadata loading duration: When the coordinator for a group changes (due to a broker restarting, for example), the new coordinator has to load the group metadata from the __consumer_offsets partitions. This can take some time, during which groups are idle. This KIP proposes adding metrics to track the duration of such reloads so it’s easy to identify when this happens and how long it takes.

Blogs

IBM Event Streams for Cloud is Apache Kafka-as-a-service for IBM Cloud.

Get started with IBM Event Streams

Mickael Maison
Software Engineer

More from Announcements
May 21, 2024

IBM Hybrid Cloud Mesh and Red Hat Service Interconnect: A new era of app-centric connectivity 

2 min read - To meet customer demands, applications are expected to be performing at their best at all times. Simultaneously, applications need to be flexible and cost effective, and therefore supported by an underlying infrastructure that is equally reliant, performant and secure as the applications themselves.   Easier said than done. According to EMA's 2024 Network Management Megatrends report only 42% of responding IT professionals would rate their network operations as successful.   In this era of hyper-distributed infrastructure where our users, apps, and data…

May 15, 2024

IBM named a Leader in Gartner Magic Quadrant for SIEM, for the 14th consecutive time

3 min read - Security operations is getting more complex and inefficient with too many tools, too much data and simply too much to do. According to a study done by IBM, SOC team members are only able to handle half of the alerts that they should be reviewing in a typical workday. This potentially leads to missing the important alerts that are critical to an organization's security. Thus, choosing the right SIEM solution can be transformative for security teams, helping them manage alerts…

May 6, 2024

IBM and MuleSoft expand global relationship to accelerate modernization on IBM Power 

2 min read - As companies undergo digital transformation, they rely on APIs as the backbone for providing new services and customer experiences. While APIs can simplify application development and deliver integrated solutions, IT shops must have a robust solution to effectively manage and govern them to ensure that response times and costs are kept low for all applications. Many customers use Salesforce’s MuleSoft, named a leader by Gartner® in full lifecycle API management for seven consecutive times, to manage and secure APIs across…

IBM Newsletters

 Get our newsletters and topic updates that deliver the latest thought leadership and insights on emerging trends.
Subscribe now More newsletters