Data compression options for Apache Kafka

You can configure Z Common Data Provider to compress data before sending it to Apache Kafka, reducing network bandwidth usage and storage requirements on Kafka brokers.

Compression types

Z Common Data Provider supports two types of compression for data sent to Kafka. You can use either compression type individually or combine both for maximum compression.
  • Kafka native compression
  • zEDC application-level compression
Kafka native compression
Kafka native compression is transport-level compression applied by the Kafka producer. The data is compressed before being sent over the network and automatically decompressed by Kafka consumers. The following algorithms are supported on z/OS®:
Algorithm Compression ratio Speed z/OS compatibility
LZ4 40-50% Extremely fast Compatible
GZIP 60-70% Slow 15-25%
zEDC application-level compression

zEDC (z/OS Enterprise Data Compression) is payload-level compression. It provides hardware‑accelerated compression on z/OS systems. When hardware acceleration is available, zEDC uses the IBM Z® z15 or later processor's integrated compression accelerator, which reduces CPU overhead.

Features:

  • Hardware acceleration on supported z/OS systems
  • Automatic fallback to software compression
  • Configurable compression levels (0-9)
  • Can be combined with Kafka native compression

Compression characteristics:

  • Compression ratio: 50-60% (hardware), 45-55% (software)
  • CPU usage: Low with hardware acceleration, moderate with software
  • Best for: z/OS environments with supported hardware

Choosing the right compression type

Use LZ4 when:

  • You need high throughput with minimal CPU overhead
  • Network bandwidth is the primary concern
  • You want the best balance of compression and speed

Use GZIP when:

  • Storage space is critical
  • You can tolerate higher CPU usage
  • Network bandwidth is severely limited

Use zEDC when:

  • Running on z/OS with IBM Java (see z/Enterprise Data Compression)
  • Hardware acceleration is available (z15 or later)
  • You want to minimize CPU overhead while achieving good compression