Kafka Connect common topologies

This section describes the three approaches that can be used when integrating IBM® MQ with Kafka through the IBM connectors.

See Obtaining the Connectors for more information on obtaining the connectors, and Using the Connectors for more information on queue manager connection and configuration options.

Direct to queue (source)

Applications which want to send data to Kafka using IBM MQ can send those messages to the queue used by the IBM MQ source connector. The IBM MQ source connector then takes those messages and transfers them to the relevant Kafka topic.

MQ application sends data using a message queue through Kafka Connect into Apache Kafka — Figure 1. Direct to queue (source)

This approach should be used when an application needs to send data into Kafka and that data is not already being sent to IBM MQ.

Sending data using IBM MQ means that the send of the message can be done inside a transaction coordinated with other updates, for example to a database. This approach also avoids the need to set up a potentially short lived connection to Kafka, and instead using an existing connection to IBM MQ.

Streaming queue copy (source)

In many cases there is a need to take a copy of existing data moving through IBM MQ and sending it into Kafka, for example, for analytics. This can be achieved using streaming queues. Streaming queues allow messages being put to one queue to be copied by the queue manager to a second queue, without affecting the applications using the first queue. For more information, see Streaming queues.

For example:

DEF QL(TO.APP) STREAMQ(TO.KAFKA) STRMQOS(MUSTDUP)
DEF QL(TO.KAFKA)

means that when a message is sent to TO.APP, a copy of that message must be sent to TO.KAFKA. The IBM MQ source connector then takes those messages from TO.KAFKA and transfers them to the relevant Kafka topic.

MQ application sends copy of data using a message queue through Kafka Connect into Apache Kafka — Figure 2. Streaming queue copy (source)

Enabling streaming queues has no effect on existing applications, as the original message does not change. The message sent to the second queue is identical to the original message with the same payload, message ID, correlation ID, and so on.

Direct to queue (sink)

As with the source connector, the sink connector can be configured to receive data from a Kafka topic into a queue directly.

Apache Kafka sends data through Kafka Connect using a message queue into an MQ application — Figure 3. Direct to queue (sink)

Receiving data through IBM MQ means that the receive of the message can be done inside an transaction coordinated with other updates, for example to a database.

This approach also avoids the need to set up a potentially short lived connection to Kafka, and instead using an existing connection to IBM MQ.

Obtaining the Connectors

[z/OS] In IBM MQ Advanced for z/OS® Value Unit Edition and IBM MQ Advanced for z/OS, the connectors and their samples are provided in the kafka-connect directory of the Connector Pack component, in z/OS UNIX System Services (USS). [MQ 9.4.3 Jul 2025] From IBM MQ 9.4.3, a supported version of the Kafka Connect framework is included.

[UNIX, Linux, Windows, IBM i] In IBM MQ Advanced for Multiplatforms and IBM MQ Appliance, these connectors and required configuration files can be retrieved by logging in to Fix Central and searching for V.R.M.F-IBM-MQ-Kafka-Connectors.tar.gz, for example, 9.4.0.0-IBM-MQ-Kafka-Connectors.tar.gz. [MQ 9.4.3 Jun 2025] From IBM MQ 9.4.3, a supported version of the Kafka Connect framework is included.

This is what is shipped with each IBM MQ version:

IBM MQ version number	IBM MQ for Multiplatforms tar file name	Source connector version	Sink connector version	XML converter version	Kafka Connect	Minimum supported Java
9.4.0	9.4.0.0-IBM-MQ-Kafka_Connectors.tar.gz	2.0.0	2.1.0	N/A	N/A	Java 8
9.4.1	9.4.1.0-IBM-MQ-Kafka_Connectors.tar.gz	2.1.0	2.2.0	N/A	N/A	Java 8
9.4.2	9.4.2.0-IBM-MQ-Kafka_Connectors.tar.gz	2.3.0	2.2.1	0.2.1	N/A	Java 8
9.4.3	9.4.3.0-IBM-MQ-Kafka_Connectors.tar.gz	2.3.0	2.2.1	0.2.1	2.13-3.9.0	Java 8
9.4.4	9.4.4.0-IBM-MQ-Kafka_Connectors.tar.gz	2.6.0	2.2.1	0.2.1	2.13-4.0.0	Java 17
9.4.5	9.4.5.0-IBM-MQ-Kafka_Connectors.tar.gz	2.7.0	2.2.1	0.2.1	2.13-4.1.0	Java 17

Notes:

You should always take the most recent version of the connectors and check regularly for updates. The connectors provided with IBM MQ are the latest at the time when the product ships, and are periodically updated to the most recent version.
If support for the IBM MQ Connectors is provided through entitlement to IBM MQ Advanced for z/OS Value Unit Edition, IBM MQ Advanced for Multiplatforms or IBM MQ Appliance, then the connectors must be connected to a queue manager running with that entitlement.

Using the Connectors

[MQ 9.4.3 Jun 2025] From IBM MQ 9.4.3, the connectors can be run using a supported version of Kafka, which is provided with the connectors.

The connectors are configured using either properties or JSON files. Sample files are provided with the connectors.

Details on the configuration options along with how to set up the connectors are provided at:

Source Connector: Kafka Connect source connector for IBM MQ
Sink Connector: Kafka Connect sink connector for IBM MQ

[MQ 9.4.0 Jun 2024] To enable exactly-once support in the source connector see Running the MQ source connector, and for the sink connector see Running the MQ sink connector. [MQ 9.4.3 Jun 2025] From IBM MQ 9.4.3, a supported version of the Kafka Connect framework is included.

[MQ 9.4.0 Jun 2024] For more information on the differences between at-least-once and exactly-once delivery, and how to configure exactly-once delivery, see Exactly once support.

For Kafka Connect to run the IBM MQ Connectors, it needs to have the connector jars, and various other jar files, on its class path. The following jar files are required:

jms.jar
com.ibm.mq.allclient.jar
org.json.jar
bcpkix-jdk18on.jar
bcprov-jdk18on.jar
bcutil-jdk18on.jar
jackson-annotations.jar
jackson-core.jar
jackson-databind.jar
slf4j-api.jar with a version >= 2.0.7

All of these other than slf4j-api.jar are already available in the IBM MQ installation. slf4j-api.jar can be downloaded from Maven.

For example:

Source connector

export CLASSPATH=$CLASSPATH:/path-to-kafka-jars/kafka-connect-mq-source-2.0.0.jar:
/path-to-mq-jars/jms.jar:/path-to-mq-jars/com.ibm.mq.allclient.jar:/path-to-mq-jars/org.json.jar:
/path-to-mq-jars/bcpkix-jdk18on.jar:/path-to-mq-jars/bcprov-jdk18on.jar:/path-to-mq-jars/bcutil-jdk18on.jar:
/path-to-mq-jars/jackson-annotations.jar:/path-to-mq-jars/jackson-core.jar:/path-to-mq-jars/jackson-databind.jar:
/path-to-slfj-jar/slf4j-api.jar

Sink connector

export CLASSPATH=$CLASSPATH:/path-to-kafka-jars/kafka-connect-mq-sink-2.1.0.jar:
/path-to-mq-jars/jms.jar:/path-to-mq-jars/com.ibm.mq.allclient.jar:/path-to-mq-jars/org.json.jar:
/path-to-mq-jars/bcpkix-jdk18on.jar:/path-to-mq-jars/bcprov-jdk18on.jar:/path-to-mq-jars/bcutil-jdk18on.jar:
/path-to-mq-jars/jackson-annotations.jar:/path-to-mq-jars/jackson-core.jar:/path-to-mq-jars/jackson-databind.jar:
/path-to-slfj-jar/slf4j-api.jar

where:

path-to-kafka-jars is the path to the location where the IBM MQ connectors are installed
path-to-mq-jars is the path to the location where the IBM JMS client is installed.
path-to-slfj-jar is the path to the location where the slf4j-api.jar is installed.

[z/OS] If running on z/OS, USS_ROOT/kafka-connect/source/kafka-connect-mq-source.jar, in the Connector Pack component, points to the most recent version of the source connector, and USS_ROOT/kafka-connect/sink/kafka-connect-mq-sink.jar points to the most recent version of the sink connector.

Kafka Connect, and the IBM MQ connectors can be run on any platform with a Java virtual machine. They do not have to run on the same platform as the queue managers, or the Kafka cluster that they connect to.

However, if there is a long distance between the queue managers and the Kafka clusters, you should position the connectors relatively close to the queue managers; ideally in the same availability zone, or data center.

Using the Connectors on z/OS

The connectors are fully supported with queue managers running on all platforms, including on z/OS. Connections to z/OS queue managers can be either through a server-connection channel, or through local bindings.

In performance testing environments on IBM z/OS and IBM MQ for z/OS, optimal performance has been obtained by running the connectors on z/OS in z/OS UNIX System Services (USS) and connecting to queue managers using local bindings. Details on these findings are available here: Kafka Connectors for IBM MQ – an MQ for z/OS perspective.

[Long Term Support] Running Kafka Connect in USS on z/OS requires some extra setup steps; documentation for these is here: Running connectors on IBM z/OS.

[MQ 9.4.3 Jul 2025] Instructions for using the connectors on z/OS with the Kafka Connect shipped in the Connector Pack from IBM MQ 9.4.3 are here: Running Kafka Connect on IBM z/OS. Alternatively instructions for manually downloading, and adjusting, Kafka Connect to run on z/OS are here: https://ibm.github.io/event-automation/es/connecting/mq/zos/.

Using the XML Converter

From IBM MQ 9.4.2, the XML Converter jar can be found at path-to-kafka-jars/kafka-connect-xml-converter-0.2.1.jar.

[z/OS] If running on z/OS, USS_ROOT/kafka-connect/xml-converter/kafka-connect-xml-converter.jar, in the Connector Pack component, points to the most recent version of the XML Converter.