JR64492: KAFKA CONNECTOR RUNNING IN CONTINUOUS MODE FAILS WITH COMMIT CANNOT BE COMPLETED ERROR.

APAR status

• Closed as Vendor Solution.

Error description

• Kafka Connector running in continuous mode, fails with Commit
  cannot be completed error.
  

Local fix

Problem summary

Problem conclusion

Temporary fix

Comments

• Problem:
  Kafka Connector running in continuous mode sometimes aborts
  reporting the following error:
  
  Fatal Error: Commit cannot be completed since the group has
  already rebalanced and assigned the partitions to another
  member. This means that the time between subsequent calls to
  poll() was longer than the configured max.poll.interval.ms,
  which typically implies that the poll loop is spending too much
  time message processing. You can address this either by
  increasing the session timeout or by reducing the maximum size
  of batches returned in poll() with max.poll.records
  
  In continuous mode, connector sends an "end of wave" marker to
  subsequent stages periodically, based on record count or time
  frame.
  The flow assumes:
  - consume a record
  - write to the link
  - mark end of wave if needed
  - commit record consumption to Kafka
  When 'end of wave' step holds execution of the code for a long
  time, it may happen that it will exceed the timeout in Kafka to
  commit consumed records. As a result, the error is reported and
  job aborts.
  
  Resolution:
  Modify Kafka Connector to use one player process with multiple
  consumer threads when reading data in continuous mode. The
  thread count equals the number of available processors by
  default.
  

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

Applicable component levels