Using WebSphere MQ Telemetry and Pachube to connect to remote sensors and devices

Pachube and the MQTT protocol with its accompanying broker implementations both offer excellent facilities for aggregating data from remote devices and sensor networks into a common API. However, they do not provide a way to integrate data between the two technologies. This article shows you how to use the new WebSphere MQ Telemetry function to bridge the two networks and share data, and includes a downloadable utility to facilitate the integration.

Share:

Nick Maynard, Senior Software Engineer, Industry and Business Solutions Team, IBM

Author photoNick Maynard is a Senior Software Engineer on the Industry and Business Solutions Team at the at the IBM Software Lab in Hursley, United Kingdom. He specializes in Dojo, AJAX, Web programming, Web services, and Linux. You can contact Nick at nick.maynard@uk.ibm.com.



01 June 2011

Also available in Chinese

Introduction

IBM® WebSphere® MQ Telemetry extends the universal messaging backbone provided by WebSphere MQ to a wide range of remote sensors, actuators, and telemetry devices. It uses the MQ Telemetry Transport (MQTT) protocol to enable you to monitor enterprise sensors using IBM MQ technology. MQTT is a messaging protocol developed by IBM, previously available in WebSphere Message Broker, and donated by IBM to the open-source community. WebSphere MQ Telemetry includes a client Java™ library to enable the development of MQTT-enabled code.

Internet of Things

According to Juniper Networks Social Business Strategist Adam Christensen, "As others have stated before, if Web 1.0 was characterized by connecting people to content, and Web 2.0 is connecting people to people, then Web 3.0 is about connecting objects to each other and to people: the Internet of Things."

The Internet of Things focuses on interconnecting and instrumenting objects in a fine-grained manner, in order to enable sophisticated and intelligent monitoring and management of complex networks of real-world objects.

Another increasingly popular technology for monitoring sensors and devices is Pachube. Pachube and MQTT are enabling technologies for the Internet of Things, and are expected to be serious players in the interconnected, instrumented world envisioned by the IBM Smarter Planet initiative.

Pachube currently has about 120,000 users (according to statshow.com), and is increasingly being used by providers such as CurrentCost to provide sensor interconnectivity. The IBM MQTT protocol is used in many diverse environments to connect a variety of sensor systems.

This article discusses the challenges of integrating the two systems and presents potential solutions, including a utility for integrating the networks. The article should be valuable to users working with the two systems or merely comparing them, and the utility should be invaluable to those integrating the two systems.

Subscribing and publishing to MQTT topics

The system design of the Pachube-MQTT bridge involves publishing information from Pachube feeds to MQTT topics, and listening on MQTT topics for information to publish to Pachube feeds, as shown below:

System design of Pachube-MQTT bridge, with mapping of Pachube feeds to MQTT topics and vice-versa
System design of Pachube-MQTT bridge

MQTT is designed from the group up to provide a publish/subscribe message model in an easy-to-understand way based on "topics." Because the protocol is openly published with a royalty-free license, many client and broker implementations exist for it. Typically, an individual source of data is mapped to an MQTT topic, which usually means that each source of sensor information will have its own topic.

Enabling technologies

The new WebSphere MQ Telemetry release includes a simple Java client, and you can use this library to both listen for updates, and to submit updates to topics.

Subscribing and publishing to Pachube feeds

When bridging MQTT and Pachube, you need to listen to Pachube feeds for information to publish onto MQTT topics, and publish information onto Pachube feeds from MQTT topics.

The Pachube API leverages the HTTP protocol to provide the interface both for publishing feed updates into the system, and for retrieving feed updates from the system. Publishing feed updates automatically is accomplished solely by pushing data over HTTP using PUT. While it is possible to "poll" for updates to feeds, the Pachube API provides a facility for registering an explicit listener, enabling Pachube to "push" updates to an application. Listeners must implement a basic subset of the HTTP protocol to be able to implement this part of the Pachube API.

Enabling technologies

Use the Apache HttpClient library for publishing updates to the Pachube system, and the Jetty lightweight Web server to implement the listener component. Additionally, Pachube pushes feed updates to listeners in a JSON envelope, so you can use the Jackson JSON library to deserialise this structure.

Integrating the networks

Any attempt to bridge the two networks must take into account the differences between the two systems as part of its design.

Throttling traffic

As a protocol, MQTT has no throttling features. However, to reduce traffic on their system, and to provide a differentiator for their premium customers, Pachube implements multiple levels of throttling for their clients. Throttling is particularly apparent in the area of publishing updates to Pachube -- basic site membership allows only three updates per minute. Therefore any integration must handle this publish throttling in an intelligent manner; naive implementations may lose updates, or fail to update feeds that receive updates less frequently. The following algorithm provides a fair solution that balances the need to provide timely updates with the need to minimise loss of information:

  1. When an update arrives from an MQTT topic, determine the target feed and add the update to an internal queue structure. If an update already exists in the queue for the target feed, replace it with the new value, but keep the original timestamp.
  2. As often as permitted by the Pachube throttling solution (if three updates per minute, then every 20 seconds), take the update with the oldest timestamp from the internal queue structure, publish it to Pachube, and remove it from the internal queue structure.

This algorithm is not perfect and certain edge cases exist. However for the majority of low-medium throughput scenarios, it should suffice. The basic utility for integration, introduced below, uses a similar algorithm.

Authentication and authorisation

Both technologies provide authentication facilities, but levels of authorisation vary.

The MQTT protocol lets brokers optionally control access using username and password pairs, but once authenticated, all actions are authorised.

Pachube allows all current data to be read by any user using the Web interface. For automatic implementations, authentication is governed using API keys. All API keys are authorised to register listeners for any feed, and API keys are authorised to publish updates to feeds by the feed owner.

A bridging utility must therefore at a minimum enable you to configure the Pachube API key on a per-feed basis.

Using the basic utility for integration

This article includes a basic utility that uses the principles described above to provide a simple integration bridge between the Pachube network and the MQTT protocol. You can download it below, extract it from the zip file, and then use the following instructions to install and configure it:

  1. Installation: Download the JAR files listed in the file lib/README.txt to run the program.
  2. Configuration: Copy config.xml.dist to config.xml, and edit it to suit your requirements.
  3. Execution: Run bridge.bat or bridge.sh according to your system type.
  4. Debugging: Edit the log4j.properties file to set per-package debugging levels.

Conclusion

This article showed you how to use the new WebSphere MQ Telemetry function to bridge the Pachube and MQTT networks and share data between them. It also introduced a basic utility to bridge the two networks.


Download

DescriptionNameSize
Code samplemqttPachubeBridge.zip30KB

Resources

  • WebSphere MQ and related resources
  • WebSphere resources
  • developerWorks resources
    • Trial downloads for IBM software products
      No-charge trial downloads for selected IBM® DB2®, Lotus®, Rational®, Tivoli®, and WebSphere® products.
    • developerWorks blogs
      Join a conversation with developerWorks users and authors, and IBM editors and developers.
    • developerWorks tech briefings
      Free technical sessions by IBM experts to accelerate your learning curve and help you succeed in your most difficult software projects. Sessions range from one-hour virtual briefings to half-day and full-day live sessions in cities worldwide.
    • developerWorks podcasts
      Listen to interesting and offbeat interviews and discussions with software innovators.
    • IBM Education Assistant
      A collection of multimedia educational modules that will help you better understand IBM software products and use them more effectively to meet your business requirements.

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into WebSphere on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=WebSphere
ArticleID=660872
ArticleTitle=Using WebSphere MQ Telemetry and Pachube to connect to remote sensors and devices
publish-date=06012011