Community

Bridging from MQ into Message Hub in IBM Bluemix

bluemix-selfieWe’re making some changes to how you can connect your MQ queue managers to cloud services in IBM Bluemix. It will take some time before the work is complete, but this post will give you a peek into what’s coming up.

For several months, we’ve had a service in Bluemix called Message Connect. This service works alongside Message Hub to take messages from sources including MQ and transfer them into Message Hub. Message Connect is an experimental service, meaning that it’s not intended for production use and doesn’t benefit from the same excellent level of support as our production services. We’d always expected it to be a temporary solution, but it turned out to be a bit less temporary than we’d anticipated. It’s time to replace it.

Message Connect provided support for streaming data from MQ and other services such as Twitter. We believe these are fundamentally different and we want to handle them in different ways. For transferring MQ messages into Message Hub, we want to provide equivalent supported functionality right inside Message Hub. For event-based integration with other services, IBM App Connect is more appropriate and provides support for a wide and growing variety of apps and services.

Message Hub Bridges

In Message Hub, we’re starting to build out a range of connectivity options collectively known as bridges. We’ve just made available a preview release of our first bridge, the Object Storage Bridge.

The Object Storage Bridge automatically takes messages from a topic and transfers them into the Object Storage service in Bluemix. If you have a source of messages that you’d like to copy into cost-effective, long-term storage, the Object Storage Bridge will be very useful. We think it’s especially good for Internet of Things data which you may like to analyze using tools such as Apache Spark.

Object Storage Bridge

The fact that it’s starting out as a preview release is most clearly visible, or perhaps not visible, in the web-based user interface. That’s the most obvious thing we’ll fix before the full release. To start, you’ll need to use a REST API to work with the Object Storage Bridge. But, for technically minded people who understand Object Storage and analytics, that’s not going to be a barrier. We’ve got some easy-to-follow instructions to get you started.

What about connecting MQ to Message Hub?

The second bridge we’re planning is a bridge from MQ into Message Hub. The MQ Bridge is the other way round. It connects to an MQ queue manager and transfers messages on an MQ queue onto a Kafka topic. The bridge runs inside Bluemix and needs to be able to connect to the queue manager. If your queue manager does not accept connections from the public internet, you can use the Secure Gateway service in Bluemix to create a tunnel which can then be used by the MQ Bridge to connect to the queue manager.

MQ Bridge

The MQ Bridge will be a fully integrated, production-ready part of Message Hub, running on our infrastructure, monitored and maintained as part of the service.

I just can’t wait – what can I do?

We’ve just deprecated the Message Connect service and will withdraw it on January 6 2017. It’s going to take a bit longer than that to get the MQ Bridge ready. So, what can you do today as an MQ customer wanting to transfer messages into Message Hub?

The most straightforward option is to write a simple Java application with two ends to run alongside your queue manager. One end connects to MQ and consumes messages from an MQ queue or topic, while the other end connects to Message Hub and publishes those messages onto a Message Hub topic. I’d recommend using Java because you’ll connect to Message Hub using an open-source client called the Apache Kafka client, and that’s in Java. For simplicity, I’d also choose the JMS API to connect to MQ.

If you’re familiar with connecting to MQ from Java, that part will be easy, but you might well not have used the Kafka client before. Luckily, we have some code to help.

An excellent starting point is the Message Hub Java console sample application. This is a simple Java application which shows how to send and receive messages using the Kafka API. The code in ProducerRunnable.java shows how to connect to Message Hub as a producer and how to send messages. With a bit of editing, that’s half the code you’ll need.

If you read the code, you’ll soon realize that the Kafka API is asynchronous. When you publish a message, the KafkaProducer.send(ProducerRecord<>) method actually returns before the message has been acknowledged. This is great for throughput because Message Hub can batch up the requests, but it means that you need to take extra care to make sure you handle any errors.

Probably the best balance of throughput and safety is to receive a bunch of messages from MQ in a transaction, publish them to Message Hub asynchronously, wait for the “Future” on the last message published, and then commit the MQ transaction if there were no errors. There are situations in which messages might be duplicated (what happens if you turn the power off, for example?), but it would be quite reliable.

Summary

Most customers using Bluemix don’t use it in isolation; it’s part of a hybrid cloud environment with some parts running in customer data centers and other parts running in cloud. Delivering a fully support capability for bridging from MQ queue managers into Message Hub is important to make messaging in the hybrid cloud a reality.

We are beginning to build out a selection of connectivity options as part of Message Hub, and expect to deliver MQ connectivity in this way over the coming months.

Share this post:

Share on LinkedIn

Add Comment
One Comment

Leave a Reply

Your email address will not be published.Required fields are marked *


Dave Arnold

Great article Andrew. I guess while we wait on the Message Hub MQ Bridge an option would be to use IBM Integration Bus 10.0.0.7 and leverage the new Kafka publish and subscribe support.

If customers don’t have IIB on-premise, they could run it up in the Bluemix container service, client connect to the MQ Queue Manager (or set up the Switch Agent) and re-publish MQ messages (or publications) on Message Hub topics.

Cheers

https://www.ibm.com/support/knowledgecenter/SSMKHH_10.0.0/com.ibm.etools.mft.doc/bz91030_.htm

Reply
More Community Stories

Fedr8 + Bluemix = Brilliant

Modernize, adopt, enhance—three words always in the mind of Rhys Sharp, CTO & Co-Founder of Fedr8. The UK-based company is a cloud enabler, which means they help clients modernize their business processes, adopt cloud technology, and enhance their results. Rhys talks about the business benefits of fully embracing the digital economy and cloud, but he knows, and his customers know, that digital transformation is no walk in the park.

Easy SSH Sessions into Bluemix

Dr. Nic Williams recently wrote a great blog post on how to run one-off command line jobs on Cloud Foundry. The same process will work on IBM Bluemix since it’s based on Cloud Foundry.

Webinar: Using Containers to Build and Deploy Scalable APIs in the Cloud

A compelling use case for exposing an API in the cloud is to make available resources of an existing enterprise application for new development. For example, let's say you have a new mobile application that awards customer loyalty points related to purchase activity. In cases of customer complaint, you may want to give service reps the option to assign a customer points as part of a satisfying resolution. To do that, you need guidelines for how many points to assign based on customer history. An existing business rules system in the company's on premises data center could provide this guidance if off premises reps could access it. In this webinar, Andrew Trice demonstrates the process of building and scaling an API on Node.js using StrongLoop's Loopback framework, whose visual editing consoles enables rapid development of complete RESTful CRUD APIs. Andrew then shows how to deploy the API onto the cloud platform in an appropriate container.