Error processing with an Apache Kafka provider

When messages have errors during processing in Kafka, you can reprocess them or mark them to be deleted after they expire from Kafka.

Error processing of messages in sequential queues

Sequential queues do not use an error queue because only one message is in error at any time. A message in error prevents the messages behind it in the queue from processing. However, an error does not prevent new messages from being added to the queue. Kafka retains both processed and unprocessed messages in the queue, based on the retention time that is set in Kafka.

In Kafka, pay attention to the Kafka retention time. For example, if you set it to 48 hours, a message goes in error late Friday evening. Other messages stack up behind the error message. If the error message is not managed until Monday morning, the error message and other unprocessed messages might expire before messages resume processing.

Messages that are in error and expired in Kafka can be reprocessed in the Message Reprocessing application.

If messages are in a queue but are not being processed, which happens if the cron task is not active, messages can expire. Because the messages did not process with an error, they are not available in the Message Reprocessing application and are not recoverable.

If you have messages in error that you do not want to reprocess, mark the messages for deletion in the Message Reprocessing application.

Error processing of messages in continuous queues

A message that is in error in a continuous queue is written to the end of the queue so that it reprocesses. The number of times that an errored message can be reprocessed is determined by the value of the Maximum Retry Count field for this external system.

For continuous inbound or outbound error processing, you must set up an instance of the KAFKAERROR cron task for that continuous queue. This cron task processes the messages from the maxinterror table in Maximo® Application Suite.

Note: After attempting the maximum retry count times, the error message is set on HOLD status. This gets skipped by the KAFKAERROR cron task in favor of the next message in the error table. If you want to fix the message and retry, use the Message Reprocessing application to edit the message and change the message status to RETRY for the KAFKAERROR cron task to process it in the next iteration. If you do not want to process the error message anymore, mark the message for deletion in the Message Reprocessing application.

Configuration of a redelivery delay for continuous queues

An administrator can configure a redelivery delay that is in effect when a retry count is specified for a continuous queue. When a redelivery delay is specified in the mxe.int.kafkaredeldelay property, the cron task for the queue does not rewrite a message to the end of the Kafka queue until after the delay time.

For example, if the delay is set to 30 seconds, the cron task for the continuous queue pauses for 30 seconds before it rewrites the message with errors to the end of the queue. The delay allows other messages to be processed in the queue, some of which can contain objects that are needed to successfully process the message that had errors. The drawback of using this mechanism is that the delay slows down the processor because it delays the cron task from continued processing.

To implement a redelivery delay, create the mxe.int.kafkaredeldelay property in the System Properties application and set a delay time in seconds.