Troubleshooting JMS destination issues

If you observe that events are flowing to the JMS queue but not to Kafka, you might need to check the JMS destination.

About this task

After you have checked that the BPM event emitter application is installed and started, and that the emitter can consume events from the JMS queue, check whether events are retried after a JMS exception. If the JMS destination setting causes error messages, check the configuration elements.

BPM event emitter cannot communicate with Kafka

About this task

You see the following error message: 
CWMCD1011E: An exception was thrown while the message was sent.
    java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.
        at org.apache.kafka.clients.producer.KafkaProducer$FutureFailure.<init>(KafkaProducer.java:1150)
        at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:846)
        at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:784)
        at com.ibm.dba.bai.events.kafka.Transport.send(Transport.java:234)
        at com.ibm.dba.bai.events.kafka.Transport.send(Transport.java:209)
        at com.ibm.dba.bai.events.kafka.Transport.send(Transport.java:193)
        at com.ibm.dba.bai.events.BusinessEventEmitter.emit(BusinessEventEmitter.java:86)
        at com.ibm.bpm.mon.oi.ConfigConnectionManager.sendToKafka(ConfigConnectionManager.java:170)
        at com.ibm.bpm.mon.oi.BPMEventEmitterMDB.onMessage(BPMEventEmitterMDB.java:133)
        [...]

This stack trace means that the BPM event emitter cannot communicate with Kafka or IBM® Event Streams to retrieve the information about the partition leader. The faulty communication prevents the Kafka cluster from sending back an acknowledgment to the BPM event emitter that rolls back the send operation and keeps the event back on the JMS queue.

Procedure

Check the following configuration elements and values.

  • In the BPMEventEmitter.yml configuration file of the BPM event emitter, check that the configuration is correct in the Deployment Manager configuration in the BPM environment.
    • In the kafka.monitor.topic section, the ingress topic name must be created on Kafka.
    • In the kafka.bootstrap.servers section, the list of bootstrap servers must be valid and use a comma as the separator. The IP addresses must be accessible from outside Kubernetes.
    • If security is enabled, the provided SSL information must be correct.
  • Check that the configuration is replicated to all the nodes of the BPM cluster.

    If not, synchronize the node with the Deployment Manager configuration and restart the BPM deployment environment.

  • Check that the Kafka brokers in the cluster are up and running, and that the minimum number of in-sync replicas server in this cluster is available.

Insufficient number of replicas in the Kafka cluster

About this task

You see the following error message:
CWMCD1011E: An exception was thrown while the message was sent.
    java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.NotEnoughReplicasException: Messages are rejected since there are fewer in-sync replicas than required.
        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:94)
        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:64)
        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:29)
        at com.ibm.bpm.mon.oi.ConfigConnectionManager.sendToKafka(ConfigConnectionManager.java:171)
        at com.ibm.bpm.mon.oi.BPMEventEmitterMDB.onMessage(BPMEventEmitterMDB.java:133)
        at com.ibm.bpm.mon.oi.BPMEventEmitterMDB.onMessage(BPMEventEmitterMDB.java:64)
        at com.ibm.ejs.container.WASMessageEndpointHandler.invokeJMSMethod(WASMessageEndpointHandler.java:138)
        at com.ibm.ws.ejbcontainer.mdb.MessageEndpointHandler.invokeMdbMethod(MessageEndpointHandler.java:1146)
        [...]

This stack trace means that the number of in-sync-replicas in the Kafka or IBM Event Streams cluster is lower than the minimum required. A wrong value prevents the Kafka cluster from sending back an acknowledgment to the BPM event emitter that rolls back the send operation and keeps the event back on the JMS queue.

Procedure

Check the following configuration elements and values.

  • Check that all the Kafka brokers are up and running.
  • If you set the kafka.min.insync.replicas Kafka producer option in the BPMEventEmitter.yml file, ensure that the value is not higher than the number of Kafka brokers in the cluster.