IBM Support

KAFKA_29 - Offset commit cannot be completed since the consumer is not part of an active group

Troubleshooting


Problem

  1. The processing time of a single batch in the pipeline is longer than the maximum polling interval (max.poll.interval.ms) in the Kafka stage.

    1. max.poll.interval.ms can be configured under Configuration Properties in the Connection tab in the Kafka Multitopic Consumer. If it is not configured, it defaults to 300000 (5 minutes) per the Kafka documentation.

    2. This causes the pipeline to not poll the consumer within the maximum interval, which causes the consumer to be removed from the consumer group, leading to the failure of the pipeline.

    3. Additionally, because the consumer has been removed from the group, it is unable to write the offset to the Kafka broker, and therefore the offset is lost (as, due to the nature of Kafka consumer groups, in Kafka pipelines, the offset is maintained by Kafka, not by SDC)

Symptom

  1. A Kafka Multitopic Consumer pipeline which is processing large amounts of data per batch fails.

  2. The pipeline is unable to retain the offset after the pipeline fails.

  3. Logs contain the following message:

    ERROR ProductionPipelineRunnable - An exception occurred while running the pipeline, com.streamsets.pipeline.api.StageException: KAFKA_29 - Error fetching data from Kafka: org.apache.kafka.clients.consumer.CommitFailedException: Offset commit cannot be completed since the consumer is not part of an active group for auto partition assignment; it is likely that the consumer was kicked out of the group.

Resolving The Problem

The processing time for any single batch of data in the pipeline must be shorter than the configured value of max.poll.interval.ms in the Kafka Multitopic Consumer. This can be accomplished in three ways:

  1. Reduce the batch size in order to reduce the processing delay.

  2. Optimize the pipeline’s data processing in order to redcue the processing delay.

  3. Configure a value for max.poll.interval.ms which is longer than the longest expected processing time for a single batch of data.

    1. For example, if the longest expected processing time for a batch of data is 10 minutes, max.poll.interval.ms must be configured to a value which is larger than 600000 (be sure to allow some margin for variance in processing times).

Document Location

Worldwide

[{"Line of Business":{"code":"LOB76","label":"Data Platform"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSM7CU","label":"IBM StreamSets Data Collector"},"ARM Category":[{"code":"","label":""}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Version(s)"}]

Document Information

Modified date:
15 March 2025

UID

ibm17186223

Page Feedback