Integration

Kafka Monthly Digest: March 2019

Share this post:

What happened in the Kafka community in March 2019?

Releases:

After three Release Candidates, Matthias J. Sax released Apache Kafka 2.2.0 on the March 26, 2019. This new minor version contains a number of interesting features:

  • The ability to re-authenticate SASL connections periodically (KIP-368)
  • Hardened inter-broker protocol (KIP-380)
  • The ability to separate Controller traffic from data plane (KIP-291)
  • Producer/AdminClient/Streams API improvements
  • Improved consumer group management (KIP-289)
  • Metrics without a value are now emitted as NaN instead of various values (KIP-386)
  • Updated kafka-topics.sh tool—it now uses the AdminClient API and does not require access to zookeeper anymore (KIP-377)
  • All command line tools now accept the --help flag (KIP-374)

As always, the full release notes are available on apache.org and the release plan is on the wiki.

KIPs:

Last month, the community submitted 10 KIPs (KIP-438 to KIP-447). These are the ones that caught my eye:

KIP-440: Extend Connect Converter to support headers
At the moment, Kafka Connect Converters don’t support message headers. In environments that rely on a lot of headers, this makes it impossible to use Connect to import or export messages. This KIP aims at fixing this discrepancy and allowing Converters to use headers.

KIP-443: Return to default segment.ms and segment.index.bytes in Streams repartition topics
In 2.0.0, the default configurations of Kafka Streams repartition topics changed in order to improve high throughput applications. However, it turns out these settings are a bit too aggressive for low throughput use cases. This KIP proposes removing these settings and instead use the default values.

KIP-444: Augment metrics for Kafka Streams
Operating Streams applications is currently relatively hard due to the lack of several key metrics. This KIP aims at addressing these issues and started doing a full review of all existing Streams metrics. The idea is to remove metrics that turned out not useful and provide new metrics to allow better monitoring and ease debugging.

Blogs:


IBM Event Streams for Cloud is Apache Kafka-as-a-service for IBM Cloud.

Get started with IBM Event Streams

Software Engineer

More Integration stories
May 2, 2019

Seamless Integration: Istio and External Services

By defining our own MCP server, we allow users to move to the Istio service mesh without any code and deployment model changes. This means we can easily use Istio to control, observe, connect, and secure services running outside Kubernetes clusters.

Continue reading

April 30, 2019

Introducing IBM Analytics Engine v1.2 and Announcing the Deprecation of IBM Analytics Engine v1.0

We are excited to inform you about the new version of IBM Analytics Engine v1.2 that will be available starting May 15, 2019. Along with this release, Analytics Engine v1.0 will be retired.

Continue reading

April 26, 2019

Help Shape the Future of Cloud Foundry

Are you a Cloud Foundry user? If so, here's your opportunity to influence the future of Cloud Foundry with the 2019 user survey.

Continue reading