December 20, 2016 | Written by: Andrew Schofield
Categorized: Community | Integration
Share this post:
We’re making some changes to how you can connect your MQ queue managers to cloud services in IBM Bluemix. It will take some time before the work is complete, but this post will give you a peek into what’s coming up.
For several months, we’ve had a service in Bluemix called Message Connect. This service works alongside Message Hub to take messages from sources including MQ and transfer them into Message Hub. Message Connect is an experimental service, meaning that it’s not intended for production use and doesn’t benefit from the same excellent level of support as our production services. We’d always expected it to be a temporary solution, but it turned out to be a bit less temporary than we’d anticipated. It’s time to replace it.
Message Connect provided support for streaming data from MQ and other services such as Twitter. We believe these are fundamentally different and we want to handle them in different ways. For transferring MQ messages into Message Hub, we want to provide equivalent supported functionality right inside Message Hub. For event-based integration with other services, IBM App Connect is more appropriate and provides support for a wide and growing variety of apps and services.
Message Hub Bridges
In Message Hub, we’re starting to build out a range of connectivity options collectively known as bridges. We’ve just made available a preview release of our first bridge, the Object Storage Bridge.
The Object Storage Bridge automatically takes messages from a topic and transfers them into the Object Storage service in Bluemix. If you have a source of messages that you’d like to copy into cost-effective, long-term storage, the Object Storage Bridge will be very useful. We think it’s especially good for Internet of Things data which you may like to analyze using tools such as Apache Spark.
The fact that it’s starting out as a preview release is most clearly visible, or perhaps not visible, in the web-based user interface. That’s the most obvious thing we’ll fix before the full release. To start, you’ll need to use a REST API to work with the Object Storage Bridge. But, for technically minded people who understand Object Storage and analytics, that’s not going to be a barrier. We’ve got some easy-to-follow instructions to get you started.
What about connecting MQ to Message Hub?
The second bridge we’re planning is a bridge from MQ into Message Hub. The MQ Bridge is the other way round. It connects to an MQ queue manager and transfers messages on an MQ queue onto a Kafka topic. The bridge runs inside Bluemix and needs to be able to connect to the queue manager. If your queue manager does not accept connections from the public internet, you can use the Secure Gateway service in Bluemix to create a tunnel which can then be used by the MQ Bridge to connect to the queue manager.
The MQ Bridge will be a fully integrated, production-ready part of Message Hub, running on our infrastructure, monitored and maintained as part of the service.
I just can’t wait – what can I do?
We’ve just deprecated the Message Connect service and will withdraw it on January 6 2017. It’s going to take a bit longer than that to get the MQ Bridge ready. So, what can you do today as an MQ customer wanting to transfer messages into Message Hub?
The most straightforward option is to write a simple Java application with two ends to run alongside your queue manager. One end connects to MQ and consumes messages from an MQ queue or topic, while the other end connects to Message Hub and publishes those messages onto a Message Hub topic. I’d recommend using Java because you’ll connect to Message Hub using an open-source client called the Apache Kafka client, and that’s in Java. For simplicity, I’d also choose the JMS API to connect to MQ.
If you’re familiar with connecting to MQ from Java, that part will be easy, but you might well not have used the Kafka client before. Luckily, we have some code to help.
An excellent starting point is the Message Hub Java console sample application. This is a simple Java application which shows how to send and receive messages using the Kafka API. The code in ProducerRunnable.java shows how to connect to Message Hub as a producer and how to send messages. With a bit of editing, that’s half the code you’ll need.
If you read the code, you’ll soon realize that the Kafka API is asynchronous. When you publish a message, the KafkaProducer.send(ProducerRecord<>) method actually returns before the message has been acknowledged. This is great for throughput because Message Hub can batch up the requests, but it means that you need to take extra care to make sure you handle any errors.
Probably the best balance of throughput and safety is to receive a bunch of messages from MQ in a transaction, publish them to Message Hub asynchronously, wait for the “Future” on the last message published, and then commit the MQ transaction if there were no errors. There are situations in which messages might be duplicated (what happens if you turn the power off, for example?), but it would be quite reliable.
Most customers using Bluemix don’t use it in isolation; it’s part of a hybrid cloud environment with some parts running in customer data centers and other parts running in cloud. Delivering a fully support capability for bridging from MQ queue managers into Message Hub is important to make messaging in the hybrid cloud a reality.
We are beginning to build out a selection of connectivity options as part of Message Hub, and expect to deliver MQ connectivity in this way over the coming months.
Sign up for Bluemix. It’s free!