Troubleshooting JMS destination issues

If you observe that events are flowing to the JMS queue but not to Kafka, also the JMS destination might need examination.

About this task

After you have checked whether the BPM event emitter application is installed and started, and that the emitter can consume events from the JMS queue, you check whether events are retried after a JMS exception. If the JMS destination setting causes error messages, you can check more configuration elements.
  • Issue 1 indicates that the BPM event emitter cannot communicate with Kafka.
  • Issue 2 indicates an insufficient number of replicas in the Kafka cluster.

Issue 1

About this task

Error message
CWMCD1011E: An exception was thrown while the message was sent.
    java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.
        at org.apache.kafka.clients.producer.KafkaProducer$FutureFailure.<init>(KafkaProducer.java:1150)
        at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:846)
        at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:784)
        at com.ibm.dba.bai.events.kafka.Transport.send(Transport.java:234)
        at com.ibm.dba.bai.events.kafka.Transport.send(Transport.java:209)
        at com.ibm.dba.bai.events.kafka.Transport.send(Transport.java:193)
        at com.ibm.dba.bai.events.BusinessEventEmitter.emit(BusinessEventEmitter.java:86)
        at com.ibm.bpm.mon.oi.ConfigConnectionManager.sendToKafka(ConfigConnectionManager.java:170)
        at com.ibm.bpm.mon.oi.BPMEventEmitterMDB.onMessage(BPMEventEmitterMDB.java:133)
        [...]
Cause
This stack trace means that the BPM event emitter cannot communicate with Kafka or IBM Event Streams to retrieve the information about the partition leader. The faulty communication prevents the Kafka cluster from sending back an acknowledgment to the BPM event emitter that rolls back the send operation and keeps the event back on the JMS queue.
Possible solutions
Check the following configuration elements and values.

Procedure

  • In the BPMEventEmitter.yml configuration file of the BPM event emitter, check that the configuration is correct in the Deployment Manager configuration in the BPM environment.
    • In the kafka.monitor.topic section, the ingress topic name must be created on Kafka.
    • In the kafka.bootstrap.servers section, the list of bootstrap servers must be valid and use a comma as the separator. The IP addresses must be accessible from outside Kubernetes.
    • If security is enabled, the provided SSL information must be correct.
  • Check that the configuration is replicated to all the nodes of the BPM cluster.

    If not, synchronize the node with the Deployment Manager configuration and restart the BPM deployment environment.

  • Check that the Kafka brokers in the cluster are up and running, and that the minimum number of in-sync replicas server in this cluster is available.

Issue 2

About this task

Error message
CWMCD1011E: An exception was thrown while the message was sent.
    java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.NotEnoughReplicasException: Messages are rejected since there are fewer in-sync replicas than required.
        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:94)
        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:64)
        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:29)
        at com.ibm.bpm.mon.oi.ConfigConnectionManager.sendToKafka(ConfigConnectionManager.java:171)
        at com.ibm.bpm.mon.oi.BPMEventEmitterMDB.onMessage(BPMEventEmitterMDB.java:133)
        at com.ibm.bpm.mon.oi.BPMEventEmitterMDB.onMessage(BPMEventEmitterMDB.java:64)
        at com.ibm.ejs.container.WASMessageEndpointHandler.invokeJMSMethod(WASMessageEndpointHandler.java:138)
        at com.ibm.ws.ejbcontainer.mdb.MessageEndpointHandler.invokeMdbMethod(MessageEndpointHandler.java:1146)
        [...]
Cause
This stack trace means that the number of in-sync-replicas in the Kafka or IBM Event Streams cluster is lower than the minimum required. A wrong value prevents the Kafka cluster from sending back an acknowledgment to the BPM event emitter that rolls back the send operation and keeps the event back on the JMS queue.
Possible solutions
Check the following configuration elements and values.

Procedure

  • Check that all the Kafka brokers are up and running.
  • If you set the kafka.min.insync.replicas Kafka producer option in the BPMEventEmitter.yml file, ensure that the value is not higher than the number of Kafka brokers in the cluster.