Kafka Connect common topologies

This section describes the three approaches that can be used when integrating IBM® MQ with Kafka through the IBM connectors.

See Obtaining the Connectors for more information on obtaining the connectors, and Using the Connectors for more information on queue manager connection and configuration options.

Direct to queue (source)

Applications which want to send data to Kafka using IBM MQ can send those messages to the queue used by the IBM MQ source connector. The IBM MQ source connector then takes those messages and transfers them to the relevant Kafka topic.

MQ application sends data using a message queue through Kafka Connect into Apache Kafka — Figure 1. Direct to queue (source)

This approach should be used when an application needs to send data into Kafka and that data is not already being sent to IBM MQ.

Sending data using IBM MQ means that the send of the message can be done inside an transaction coordinated with other updates, for example to a database. This approach also avoids the need to set up a potentially short lived connection to Kafka, and instead using an existing connection to IBM MQ.

Streaming queue copy (source)

In many cases there is a need to take a copy of existing data moving through IBM MQ and sending it into Kafka, for example, for analytics. From IBM MQ 9.3 this can easily be achieved using streaming queues. Streaming queues allow messages being put to one queue to be copied by the queue manager to a second queue, without affecting the applications using the first queue. See Streaming queues for more information.

For example:

DEF QL(TO.APP) STREAMQ(TO.KAFKA) STRMQOS(MUSTDUP)
DEF QL(TO.KAFKA)

means that when a message is sent to TO.APP, a copy of that message must be sent to TO.KAFKA. The IBM MQ source connector then takes those messages from TO.KAFKA and transfers them to the relevant Kafka topic.

MQ application sends copy of data using a message queue through Kafka Connect into Apache Kafka — Figure 2. Streaming queue copy (source)

Enabling streaming queues has no effect on existing applications, as the original message does not change. The message sent to the second queue is identical to the original message with the same payload, message ID, correlation ID, and so on.

Direct to queue (sink)

As with the source connector, the sink connector can be configured to receive data from a Kafka topic into a queue directly.

Apache Kafka sends data through Kafka Connect using a message queue into an MQ application — Figure 3. Direct to queue (sink)

Receiving data through IBM MQ means that the receive of the message can be done inside an transaction coordinated with other updates, for example to a database.

This approach also avoids the need to set up a potentially short lived connection to Kafka, and instead using an existing connection to IBM MQ.

Obtaining the Connectors

The version of the connectors that are shipped with IBM MQ change over time. The version 1 connectors that are shipped with IBM MQ 9.3.3 provide at-least-once message delivery.

[MQ 9.3.4 Oct 2023] The version 2 connectors that are shipped from IBM MQ 9.3.4 provide at-least-once, and exactly-once, message delivery.

[MQ 9.3.4 Oct 2023] For more information on the differences between at-least-once and exactly-once delivery, and how to configure exactly-once delivery, see Exactly once support.

In IBM MQ Advanced for z/OS® Value Unit Edition, [Long Term Support] [MQ 9.3.4 Oct 2023] and from IBM MQ 9.3.4 in IBM MQ Advanced for z/OS, or Long Term Support with APAR PH56722 applied, the connectors and their samples are provided in the kafka-connect directory of the Connector Pack component, in z/OS UNIX System Services (USS).

In IBM MQ Advanced for Multiplatforms and IBM MQ Appliance, these connectors and required configuration files can be retrieved by logging in to Fix Central and searching for V.R.M.F-IBM-MQ-Kafka-Connectors.tar.gz, for example, 9.3.3.0-IBM-MQ-Kafka-Connectors.tar.gz.

These are the connectors shipped with each IBM MQ version:

IBM MQ version number	IBM MQ for Multiplatforms tar file name	Source connector version	Sink connector version	Exactly-once delivery support
9.3.3	9.3.3.0-IBM-MQ-Kafka_Connectors.tar.gz	1.3.2	1.5.0	No
9.3.4	9.3.4.0-IBM-MQ-Kafka_Connectors.tar.gz	2.0	2.0	Yes

You can also obtain the version 1 connectors through the release pages from GitHub:

Source: Kafka-connect-mq-source
Sink: Kafka-connect-mq-sink

Note that any version obtained from these links is supported by IBM.

Notes:

However, you should always take the most recent version of the connectors and check regularly for updates. The connectors provided with IBM MQ are the latest at the time when the product ships, and are periodically updated to the most recent version.
If support for the IBM MQ Connectors is provided through entitlement to IBM MQ Advanced for z/OS Value Unit Edition, IBM MQ Advanced for Multiplatforms, or IBM MQ Appliance, then the connectors must be connected to a queue manager running with that entitlement.
Versions of IBM MQ Advanced for z/OS Value Unit Edition, IBM MQ Advanced for Multiplatforms, or IBM MQ Appliance that are earlier than IBM MQ 9.3.3 also provide entitlement for the IBM MQ Connectors, but the connectors must be downloaded from the release pages in GitHub, referenced in the preceding text.
Continuous Delivery versions of IBM MQ Advanced for z/OS that are earlier than IBM MQ 9.3.4 also provide entitlement for the IBM MQ Connectors, but the connectors must be downloaded from the release pages in GitHub, referenced in the preceding text.

Using the Connectors

The connectors are configured using either properties or JSON files. Sample files are provided with the connectors.

Details on the configuration options along with how to set up the connectors are provided at:

Source Connector: Kafka Connect source connector for IBM MQ
Sink Connector: Kafka Connect sink connector for IBM MQ

[MQ 9.3.4 Oct 2023] To enable exactly-once support in the source connector see Running the MQ source connector, and for the sink connector see Running the MQ sink connector.

[MQ 9.3.4 Oct 2023] For more information on the differences between at-least-once and exactly-once delivery, and how to configure exactly-once delivery, see Exactly once support.

For Kafka Connect to run the IBM MQ Connectors, it needs to have the connector jars, and various IBM MQ jar files, on its class path. The following jar files are required:

jms.jar
com.ibm.mq.allclient.jar
org.json.jar
bcpkix-jdk18on.jar (From IBM MQ 9.3.5)
bcpkix-jdk15to18.jar (IBM MQ 9.3.3 and IBM MQ 9.3.4)
bcprov-jdk18on.jar (From IBM MQ 9.3.5)
bcprov-jdk15to18.jar (IBM MQ 9.3.3 and IBM MQ 9.3.4)
bcutil-jdk18on.jar (From IBM MQ 9.3.5)
bcutil-jdk15to18.jar (IBM MQ 9.3.3 and IBM MQ 9.3.4)

For example:

Source connector

From IBM MQ 9.3.5:

export CLASSPATH=$CLASSPATH:/path-to-kafka-jars/kafka-connect-mq-source-1.3.2.jar:
/path-to-mq-jars/jms.jar:/path-to-mq-jars/com.ibm.mq.allclient.jar:/path-to-mq-jars/org.json.jar:
/path-to-mq-jars/bcpkix-jdk18on.jar:/path-to-mq-jars/bcprov-jdk18on.jar:/path-to-mq-jars/bcutil-jdk18on.jar

IBM MQ 9.3.3 and IBM MQ 9.3.4

export CLASSPATH=$CLASSPATH:/path-to-kafka-jars/kafka-connect-mq-source-1.3.2.jar:
/path-to-mq-jars/jms.jar:/path-to-mq-jars/com.ibm.mq.allclient.jar:/path-to-mq-jars/org.json.jar:
/path-to-mq-jars/bcpkix-jdk15to18.jar:/path-to-mq-jars/bcprov-jdk15to18.jar:/path-to-mq-jars/bcutil-jdk15to18.jar

Sink connector

From IBM MQ 9.3.5:

export CLASSPATH=$CLASSPATH:/path-to-kafka-jars/kafka-connect-mq-sink-1.5.0.jar:
/path-to-mq-jars/jms.jar:/path-to-mq-jars/com.ibm.mq.allclient.jar:/path-to-mq-jars/org.json.jar:
/path-to-mq-jars/bcpkix-jdk18on.jar:/path-to-mq-jars/bcprov-jdk18on.jar:/path-to-mq-jars/bcutil-jdk18on.jar

IBM MQ 9.3.3 and IBM MQ 9.3.4

export CLASSPATH=$CLASSPATH:/path-to-kafka-jars/kafka-connect-mq-sink-1.5.0.jar:
/path-to-mq-jars/jms.jar:/path-to-mq-jars/com.ibm.mq.allclient.jar:/path-to-mq-jars/org.json.jar:
/path-to-mq-jars/bcpkix-jdk15to18.jar:/path-to-mq-jars/bcprov-jdk15to18.jar:/path-to-mq-jars/bcutil-jdk15to18.jar

where:

path-to-kafka-jars is the path to the location where the IBM MQ connectors are installed
path-to-mq-jars is the path to the location where the IBM JMS client is installed.

[z/OS] If running on z/OS, USS_ROOT/kafka-connect/source/kafka-connect-mq-source.jar, in the Connector Pack component, points to the most recent version of the source connector, and USS_ROOT/kafka-connect/sink/kafka-connect-mq-sink.jar points to the most recent version of the sink connector.

Kafka Connect, and the IBM MQ connectors can be run on any platform with a Java Virtual machine. They do not have to run on the same platform as the queue managers, or the Kafka cluster that they connect to.

However, if there is a long distance between the queue managers and the Kafka clusters, you should position the connectors relatively close to the queue managers; ideally in the same availability zone, or data center.

Using the Connectors on z/OS

The connectors are fully supported with queue managers running on all platforms, including on z/OS. Connections to z/OS queue managers can be either through a server-connection channel, or through local bindings.

In performance testing environments on IBM z/OS and IBM MQ for z/OS, optimal performance has been obtained by running the connectors on z/OS in z/OS UNIX System Services (USS) and connecting to queue managers using local bindings. Details on these findings are available here: Kafka Connectors for IBM MQ – an MQ for z/OS perspective.

Running Kafka Connect in USS on z/OS requires some extra setup steps; documentation for these is here: Running connectors on IBM z/OS.