End-to-end integration with pervasive messaging and IBM Lotus Expeditor micro broker

Micro broker, a new feature in IBM Lotus Expeditor V6.1, provides connectivity services beyond enterprise servers to mobile and pervasive devices. Learn about Lotus Expeditor micro broker and how you can enable true end-to-end integration in a Service-Oriented Architecture.

Share:

Martin Gale (martin_gale@uk.ibm.com), Senior IT Specialist, Master Inventor, Lead Developer of the Lotus Expeditor micro broker, IBM

Martin Gale is the lead developer of the Lotus Expeditor micro broker and is an appointed IBM Master Inventor. Martin works in the IBM UK Software Development Laboratory in Hursley. Martin's background is in exploitation of emerging technologies for the Web and pervasive channels. Prior to his work in software development, Martin worked in a services role working with IBM's customers to develop first-of-a-kind solutions using new technologies. Martin holds a first class BSc degree in Computer Science from Portsmouth University for which he was sponsored by IBM. You can reach Martin at martin_gale@uk.ibm.com.



30 May 2007

Also available in Chinese

One of the fundamental requirements of a Service-Oriented Architecture (SOA) is connectivity. Connectivity between enterprise-class applications and services can be realized using products such as IBM WebSphere MQ and IBM WebSphereMessage Broker. Adding IBM Lotus Expeditor micro broker, a new feature of Lotus Expeditor V6.1, to the solution enables true end-to-end integration in an SOA by providing connectivity services to mobile and pervasive devices. This article introduces the key concepts of messaging and of Lotus Expeditor micro broker and the value that can be gained in a solution by using it. Learn how to design business integration solutions that extend the reach of the integration layer out into the very edges of the enterprise on mobile or embedded devices.

Introduction to messaging

Messaging middleware software, such as IBM WebSphere MQ, provides connectivity between networked applications and services. A middleware layer provides a platform for the deployment of robust, intercommunicating applications, separate from the concerns of exactly how the communication is achieved. Therefore, the middleware layer enhances the focus on the core business function of the application. Middleware promotes loose coupling between applications and components, resulting in a more flexible architecture.

In messaging middleware, connectivity is achieved by passing data between systems through a server component known as a message broker (or a queue manager) in the form of messages. Messages are packets of application data that typically consist of a data payload and configurable header properties associated with the message. The message broker or queue manager determines which clients should receive which messages based on a particular destination described in the message, and it delivers the messages accordingly. Message delivery can be synchronous, where the client application explicitly requests the next message (client pull), or asynchronous, where the broker sends the message as soon as it is matched to the client (server push). Two paradigms are used to describe destinations in messaging systems:

  • The point-to-point messaging paradigm
  • The publish-and-subscribe messaging paradigm

Point-to-point messaging

Point-to-point messaging is a means of achieving one-to-one delivery semantics between two applications. Destinations in point-to-point messaging are defined as queues. In simple terms, one application puts a message to a given queue in the broker or queue manager, and another application reads the message from the queue as shown in figure 1. The messages are delivered from the sender to the receiving application using a first-in-first-out (FIFO) mechanism.

Figure 1. Message delivery in the point-to-point messaging paradigm
Message delivery in the point-to-point messaging paradigm

A retail outlet is an example of the point-to-point messaging paradigm, in which a queue of orders is created by a store with each order being taken from the queue at the store’s warehouse so the goods can be delivered.

Publish-and-subscribe messaging

Publish-and-subscribe messaging is a way to achieve one-to-many delivery semantics between applications for a given message as shown in figure 2. A publish-and-subscribe message destination is known as a topic. A topic provides a hierarchical naming structure so that applications (subscribers) can register varying degrees of interest. A topic is a character string with each level in the hierarchy delimited in some fashion (often by a slash), for example, stocks/IBM. When sending a message, the sending party (the publisher) specifies a target topic. The broker matches an inbound message to any subscribers using this topic and delivers a copy of the message to each matching subscriber. The match may be exact if the publisher and subscriber specify exactly the same topic name, or the subscriber may specify a wildcard to match many different published topics. For example, subscribers to stocks/+ receive messages published to any topics specifying two levels of hierarchy with the first level as stocks (so messages published to stocks/IBM and stocks/MyCorp would both be matched).

Figure 2. Message delivery in the publish-and-subscribe messaging paradigm
Message delivery in the publish-and-subscribe paradigm

The publish-and-subscribe messaging paradigm is particularly suited to event-driven systems in which many other applications respond to a given trigger, such as a sensor sending a reading (such as temperature) and a variety of display units and backend systems receiving notification of the result.

Quality of service and persistence

In messaging, quality of service is concerned with the method of delivery of the data from one application to another and recovery of the application data in the event of a failure of some kind. Choosing the appropriate quality of service is often a trade-off between the volatility and relative importance of the data and the amount of system resources consumed to achieve the desired quality of service. A high quality of service typically consists of a series of exchanges between the application client and the message broker to ensure, as far as possible, that a message is delivered once and only once. Lower qualities of service often involve fewer exchanges between client and message broker (with associated gains in resource and network utilization), but they offer fewer guarantees regarding the delivery of the message, such as increased chances of duplication or loss of messages.

Persistence of the message is a similar trade-off; it is concerned with the longevity of the message when it has reached the message broker. Persistent messages are written to disk and hence, may be recovered if the message broker crashes for some reason. Non-persistent messages live only in the memory of the message broker, and in the event of a crash they most likely are lost. As with quality of service, the trade-off is one of system resources (in this case, the disk) and the latency associated with a disk I/O for each message against the ability to recover a message if a problem occurs with the broker. Financial applications typically use a high quality of service and persistent messages because of the critical nature of their data. Applications in which data is frequently updated, has relatively low cost associated with each message, and/or is deployed on a large scale may use a lower quality of service. An Internet sports scoreboard is one example of this sort of application.

Messaging clients, JMS, and JNDI

In order for applications to communicate using the message broker, they use a messaging client library. Messaging middleware vendors typically offer a variety of client libraries that support particular application platforms. Some of these client libraries offer proprietary application programming interfaces (APIs) specific to the vendor. Messaging client libraries typically expose the notion of a connection to the message broker and a number of associated verbs, such as send, receive, connect, and disconnect. Client libraries usually also include messaging-specific data types that define the entities in the middleware such as messages, queues, and topics; they also usually include exception handling routines to allow the application to handle a failure scenario.

For Java, J2EE defines the Java Messaging Service (JMS) standard, which specifies a standard set of Java interfaces and behavioral semantics to which vendors can develop Java client libraries. This standard enables applications to be written to the standard JMS interfaces and to expect certain behavior from the client without depending upon any given software vendor’s product. If the JMS provider supports the use of a Java Naming and Directory Interface (JNDI) provider, it is possible to decouple the application even further from its administered runtime environment by having named messaging resources stored as administered objects in the JNDI repository instead of the application having to hard-code references to queues and topics in the code. In this way, if the administrator of the J2EE application changes the name of a queue, for example, as long as the administered object is stored using the same name in the JNDI repository, the application does not need to be changed. Many enterprise messaging vendors support this capability, including IBM WebSphere MQ, IBM WebSphere Process Server, and IBM WebSphere Enterprise Server Bus as well as the IBM Lotus Expeditor micro broker.


Pervasive messaging and the micro broker

Typically, messaging middleware is designed for deployment on an enterprise infrastructure, usually a combination of a number of large servers and a plentiful supply of network bandwidth. This means that the enterprise application can cope with the demands of a large-scale application such as a banking system. Since the mid- to late-1990s, the phenomenon of pervasive computing has meant that computing capability has become increasingly available on mobile devices (such as PDAs and mobile telephones) and in non-computing domains (such as in cars and home appliances). The enhanced capability of these solid-state devices, therefore, provides more possibilities in terms of their participation in enterprise Service-Oriented Architectures (SOAs). Pervasive messaging enables people, applications, and devices to share information and to respond to each other regardless of where they are located in relation to each other.

Supervisory Control and Data Acquisition (SCADA) and telemetry applications are a good example of device-to-device communication whereby information from remote locations (such as power stations, oil refineries, and so on) can be obtained and fed into an enterprise backend to drive better and more timely decision making based on information from the field. Device-to-device communication also assists with decisions made within the enterprise to affect changes at the remote location (opening or closing a valve or stopping an appliance, for example). Pervasive messaging can integrate people with devices in a mobile workforce where representatives are not office-based and are working with mobile devices rather than with desktop or laptop computers. Examples of this are the delivery recording applications used by dispatch drivers, order entry systems from field sales representatives, and so on.

The pervasive domain provides a number of challenges for those wishing to provide solutions in such scenarios. First of all, devices are just that – devices, not dedicated computers. Their primary role is not computation, and often they have strict requirements regarding their physical scale (as is the case with handheld devices). This naturally places restrictions on them as a computing platform; compute resources tend to be limited, in terms of processing power, memory, and persistent storage. Also, in many cases, connectivity to the device may be available only over slow, unreliable, or expensive networks, often a combination of all three. Pervasive messaging provides middleware capabilities that are optimized for the special demands of this domain. The Lotus Expeditor micro broker aims to meet the requirements of these scenarios by providing the following specialist capabilities for the pervasive domain:

  • A small footprint message broker (the micro broker itself) to act as an integration hub in a resource-constrained environment and to link to the enterprise backend
  • Application client libraries designed to run in a variety of platforms from J2ME to J2SE to J2EE (JMS and JNDI) using an optimized pervasive messaging protocol
  • A bridge into enterprise messaging servers at the backend

The IBM Lotus Expeditor micro broker

As its name suggests, the IBM Lotus Expeditor micro broker is, in fact, a very small footprint implementation of a publish-and-subscribe message broker server. In contrast to an enterprise message broker (such as IBM WebSphere Message Broker), the micro broker is designed to be run on a device rather than on a large-scale server infrastructure. The micro broker is implemented in Java and runs on the desktop and device runtimes provided by Lotus Expeditor as well as a J2SE Java runtime. Client applications can connect to the micro broker either remotely through TCP/IP or from within the same Java runtime. The Lotus Expeditor Toolkit provides the development platform and toolset for development and testing of micro broker client applications.

The micro broker itself can either run in standalone mode or bridge into an enterprise messaging server. Within this bridge, users can define transformation logic to modify (or even discard) messages flowing between the edge and the enterprise.

Typical usage patterns for the micro broker include the following:

  • As a local integration hub on a remote site to enable on-site applications to communicate easily in a loosely coupled fashion without interapplication dependencies on proprietary device interfaces.
  • As a means of collecting remote data at the edge of the enterprise and selectively transferring it into the backend. An example of this is in a Radio-Frequency ID (RFID) scenario whereby each tag passing through a reader may generate several read events when only a single event should be communicated to the enterprise.
  • As an interprocess communication component either within a given Java runtime or across several Java and non-Java runtimes.

The micro broker and its related components build on OSGI technology and are designed to run within an OSGI runtime. This open platform promotes a service-oriented model for developing device applications. The micro broker and its client libraries are packaged in a set of OSGI bundles with client and administrative applications able to exploit the modular OSGI framework.

Administering a micro broker

The micro broker is administered using a programmatic Java programming interface. The administrative interface enables users to query various aspects of a given micro broker instance and its connected clients. Users can obtain object-oriented representations of the internals of the micro broker (such as its publish-and-subscribe matching engine) and query for information, such as the number of connected clients and the number of messages sent and received.

The core micro broker bundle exposes a BrokerFactory service to the OSGI runtime that bundles can track using the OSGI Service Tracker. An example of how this is done is shown in listing 1.

Listing 1. Using the micro broker administrative API
public class TestActivator implements BundleActivator, ServiceTrackerCustomizer {

	ServiceTracker tracker = null;
	BundleContext bundleContext = null;
	
	public void start(BundleContext context) throws Exception {
		// Keep the reference to the bundle context
		bundleContext = context;
		// Track for the Broker Factory service
tracker = new ServiceTracker(context,
“com.ibm.micro.admin.BrokerFactory",
this);
// Will call us back on addingService()
		tracker.open();
	}

	…

	public Object addingService(ServiceReference reference) {
		BrokerFactory factory = 
			(BrokerFactory)bundleContext.getService(reference);
		try {
			// Get a reference to the TestBroker instance
			// by name as we are in the same JVM
			LocalBroker broker = factory.getByName("TestBroker");
			// Simply print out the number of bytes received
			// by the broker
			System.out.println(broker.getBytesReceived());

			// We could also get by address for a remote broker
			RemoteBroker remoteBroker = 
factory.getByAddress("tcp://localhost:1883");
			remoteBroker.logon("Admin", "password"); 
// Simply spit out the number of connected clients
			System.out.println(remoteBroker.getCommunications().
getNumberOfClients());
			remoteBroker.logoff();

		} catch (AdminException e) {
			e.printStackTrace();
		}						
		return f;
	}

}

Micro broker clients

The micro broker provides two client libaries, a JMS and a proprietary client designed for use in specialist device domains. Both clients support the publish-and-subscribe messaging paradigm.

The JMS client is recommended for developing applications in the Lotus Expeditor environment. The micro broker’s JMS client implementation supports the JNDI administration model, and therefore, its administered objects can be integrated into a J2EE runtime as a JNDI JMS provider. The micro broker’s JMS client supports the publish-and-subscribe domain of JMS with the following limitations:

  • Only JMS Text and Bytes messages are supported.
  • No support is provided for distributed (XA) transactions.

As per the JMS specification, the micro broker’s JMS client allows messaging applications to receive messages from the micro broker either synchronously or asynchronously through a listener. The micro broker JMS client supports both the persistent and non-persistent flavors of quality of service defined in JMS. Listing 2 shows a simple micro broker JMS client application.

Listing 2. An example of a micro broker client application using the JMS API
JmsFactoryFactory jmsFactory = JmsFactoryFactory
	.getInstance(MQTTConstants.PROVIDER_NAME);

// Create a micro broker topic connection factory
TopicConnectionFactory connFactory = jmsFactory
	.createTopicConnectionFactory();
// Set the URI telling the factory how to connect to the server
((JmsConnectionFactory) connFactory).setStringProperty(
		MQTTConstants.MQTT_CONNECTION_URL,
		MQTTConstants.MQTT_TCP_SCHEMA + "127.0.0.1:1883");
// Create a connection object
TopicConnection conn = connFactory.createTopicConnection();
// Must set a client ID
conn.setClientID("jmsClient");
// Start the connection (connect).
conn.start();
// Create a transacted JMS session. A transacted session means
// we have to explicitly commit any sent or received messages.
TopicSession session = conn.createTopicSession(true,
		Session.SESSION_TRANSACTED);
Topic topic = session.createTopic("test");

// Create a message consumer to receive messages on the topic 
// we just created
MessageConsumer consumer = session.createConsumer(topic);

// Send 10 text messages and receive them using our consumer
for (int i = 0; i < 10; i++) {
	TextMessage message = session.
createTextMessage("Test " + (i + 1));
	MessageProducer producer = session.createProducer(topic);
producer.send(message);
	session.commit();
	producer.close();
			
	TextMessage r = (TextMessage) consumer.receive();
	session.commit();

	System.out.println("Received: " + r.getText());
}

// Close the consumer and session
consumer.close();
session.close();
		
// Now tidy up the connection
conn.close();

The micro broker supports the specialist pervasive messaging MQTT protocol specification and provides a proprietary Java MQTT messaging client. For more information on the MQTT specification see the Web site http://www.mqtt.org. This client is supplied for use in specialist device domains. The MQTT client has much simpler semantics than JMS because there is no concept of sessions, consumers, producers, or application control of transactions. Application messages are delivered asynchronously through a simple callback mechanism and are provided only in byte array payload format. MQTT messages also do not have the concept of user-defined header properties as JMS does. The MQTT client programming model also includes as a component of its callback mechanism a notification method to indicate to the application client that the connection to the broker has been lost such that the application can decide whether or not to reconnect.

The MQTT client supports the three qualities of service defined by the MQTT protocol specification for sending messages to the broker:

  • 0 – Messages are delivered at most once and not persisted to disk within the broker.
  • 1 – Messages are delivered at least once (that is, they may be duplicated) and are persisted to disk within the broker.
  • 2 – Messages are delivered once and only once and are persisted to disk within the broker.

Note that subscribing MQTT clients specify a receiving quality of service on each subscription – when publications are matched to subscribers and the subscription is at a lower quality of service, the message is downgraded to the subscribing quality of service. A simple example of an MQTT application is shown in listing 3.

Listing 3. An example of a micro broker client application using the MQTT client
// Set any specific connection properties using a properties object
MqttProperties mqttProps = new MqttProperties();
// Stateless client -- clear up any persisted data for the client
mqttProps.setCleanStart( true );
// Create the client from the factory
// The client id for this client is "testClient" and the URL in the 
// second
// parameter describes the location of the remote broker. 
// In this case we are
// connecting over TCP/IP to a broker on the host "mybroker" on 
// port 1883.
MqttClient client = MqttClientFactory.INSTANCE.createMqttClient("testClient", 
				brokerURL, mqttProps);
// Connect to the broker
client.connect();
// Publish a message to the a/b topic at QoS 2
client.publish("a/b", new MqttPayload(("Hello World").getBytes(),0), (byte) 2 , false);
// Then disconnect
client.disconnect();

NOTE: The MQTT protocol is also supported by the IBM WebSphere Message Broker V6.

The micro broker bridge

The micro broker bridge is a key component that provides the linkage between the pervasive messaging domain at the edge and the enterprise backend. As shown in figure 3, the micro broker bridge provides connectivity into a remote message broker in three flavors:

  • Through the MQTT protocol to connect either another micro broker or a WebSphere Message Broker server
  • Through JMS to connect to a WebSphere MQ queue manager
  • To a third-party JMS provider using JNDI administered objects
Figure 3. An example of a micro broker bridge topology
An example of a micro broker bridge topology

The bridge defines the following concepts to link the micro broker to a remote peer:

  • A pipe representing the bidirectional connection between the micro broker and its peer.
  • An end-point specific connection, such as a JNDI connection factory within a JNDI repository. The connection is associated with a given pipe.
  • Flows representing the message stream between a micro broker and its remote peer. A pipe may define an inbound flow (receiving messages into the micro broker) and an outbound flow (sending messages out of the micro broker). Each flow defines a set of source destinations (for example, a set of topics and queues) and a target destination to which the messages are to be delivered.
  • A series of transformations that may be defined on each flow. Transformations can modify or discard a message as it passes over the bridge.

The bridge enables the micro broker to be deployed in a broad variety of topologies, such as a set of interlinked micro brokers or a simple link between a local site to a remote backend. The ability to modify and discard messages provides a powerful mechanism by which the micro broker can isolate local application logic and concerns from the enterprise backend such that only relevant information is sent back and in a format such that it can be consumed by the enterprise messaging server as easily as possible.

The bridge is configured using the micro broker’s administrative programming interface. A simple pipe over an MQTT connection to another micro broker can be defined as shown in listing 4, using the micro broker administration interface.

Listing 4. Using the micro broker administrative API to configure the bridge
// Obtain a reference to the bridge from the administrative broker 
// object
LocalBroker broker = factory.getByName(“TestBroker”);
Bridge bridge = broker.getBridge();

// Create a definition for a Pipe		
PipeDefinition pipeDef = bridge.createPipeDefinition("anMQTTPipe");
	
// Connection definitions describe the endpoint of the pipe
// and are target-specific.	
MQTTConnectionDefinition swtest = 
bridge.createMQTTConnectionDefinition("swtest");
swtest.setHost("mybrokerserver.com");
swtest.setPort(1883);
pipeDef.setConnection(swtest);

// Create a message flow over the pipe
FlowDefinition flow1 = bridge.createFlowDefinition("flow1");
// This will set the source to a topic named “outbound” – this is where
// messages will be consumed from.
flow1.setSources(new 
TopicDefinition[]{ bridge.createTopicDefinition("outbound") });

// Set the target (i.e. where messages will be sent to) to a topic
// named “inbound” on the remote broker
flow1.setTarget(bridge.createTopicDefinition("inbound"));

// Sets the flow as the outbound flow to send messages out of the given
// micro broker into a remotely bridged broker as described above.
pipeDef.addOutboundFlow(flow1);

// Add the pipe to the bridge 
bridge.addPipe(pipeDef);
// Start all the configured pipes.
bridge.startAllPipes();

Conclusion

Messaging middleware provides a cleanly separated platform for robust and reliable intercommunication between application services and components. Where enterprise messaging provides this capability for large-scale backend systems, pervasive messaging broadens the reach of the enterprise system by providing specialist protocols and components to link in remote devices with the enterprise. The Lotus Expeditor micro broker and its clients provide an integration hub that can be deployed in resource-constrained pervasive devices and that can connect back into the enterprise through its bridge feature. Within the bridge, messages can be modified (and even discarded) such that even the link into the enterprise can be performed in the most efficient way possible. In this way, the combination of Lotus Expeditor micro broker and an enterprise messaging infrastructure enables true end-to-end integration within an SOA solution.

Resources

Learn

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into IBM collaboration and social software on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Lotus, WebSphere
ArticleID=227015
ArticleTitle=End-to-end integration with pervasive messaging and IBM Lotus Expeditor micro broker
publish-date=05302007