Using workload management, service discovery, and the Integration Registry in IBM Integration Bus V9

This article shows you how to use new service discovery and workload management features in IBM Integration Bus V9 to discover services by interrogating WebSphere MQ or a database, use the discovered service to configure integration flows, and control application behavior at runtime. You will also learn how to use the Integration Registry to store and manage the discovered services and workload management policies.

Share:

Eugene Kharlamov (eugenek@ca.ibm.com), Software Developer, IBM Integration Bus Development Team, IBM

Photo of Eugene KharlamovEugene Kharlamov is a Software Developer on the IBM Integration Bus Development Team. He has 15 years of experience with IBM in product development and consultant roles for different WebSphere products. You can contact Eugene at eugenek@ca.ibm.com.



John Reeve (jreeve@uk.ibm.com), Software Developer, IBM Integration Bus Development Team, IBM

Photo of John ReeveJohn Reeve is a Software Developer on the IBM Integration Bus Development Team. He has 15 years of experience with IBM in diverse roles from testing to development. He has been a Lead Developer in many key areas in recent releases of WebSphere Message Broker and IBM Integration Bus, including support for TCP/IP, file processing, IMS, sequencing, aggregation, workload management, ESQL, Connect:Direct, WebSphere MQ FTE, and SAP, Siebel, and PeopleSoft adapters. You can contact John at jreeve@uk.ibm.com.



Sarah Styles (sarah.styles@uk.ibm.com), Software Developer, IBM Integration Bus Development Team., IBM

Photo of Sarah StylesSarah Styles is a Software Developer on the IBM Integration Bus Development Team. She has seven years of experience with IBM and has worked in Test, Development, and Support roles for WebSphere Message Broker and IBM Integration Bus. Currently she works on IBM Integration Bus web administration as well as workload management and policy. You can contact Sarah at sarah.styles@uk.ibm.com.



18 September 2013

Introduction

IBM® Integration Bus V9 provides three new features to improve integration application development and runtime management:

1. WebSphere MQ and database service discovery

You can now discover WebSphere® MQ and database services from live systems using the IBM Integration Bus Toolkit. Representing an MQ interaction as a service with well-defined interface, binding information, and message model structure simplifies the configuration of integration flows. You can use the same service to build an integration web service facade for the MQ service or database integration. The service description is created by connecting and interrogating an IT system to discover definitions such as data definitions, function declarations, or communication objects. Once discovered, the service description can be used to configure the message flow nodes implementing the service description, and shared with different tools and users. You can:

  • Understand the service interface describing the logical description of available business objects and operations.
  • Use system-specific connection configuration and physical exchange formats.

A service description can be used to provide or consume a service:

  • Providing a service requires a service description and implementation (message flows).
  • Consuming a service from an integration requires a complete service description and configured access method (message flow nodes).

Two discoverable service types are supported:

  • MQ Service -- Discover MQ services created using MQ resources from the target system. This article describes the details of the MQ service discovery and usage.
  • Database service -- Discover database services using a definition file (.DBM).

2. Workload management (WLM)

WLM enables you to manage message flow workload as configured by a policy.

3. Publishing MQ service descriptions and WLM policies to the Integration Registry

The Integration Registry is hosted by the integration node and facilitates collaboration and reuse. Discovered services and WLM policies are stored and accessed in the Integration Registry.

Service discovery

Previously, message flow nodes interacting with MQ or databases had to be manually configured -- a tedious and error-prone task. Now you can discover the MQ service by introspection of the live MQ system, and configure the service input and output messages using XML schemas. Once the discovery is complete, the service description is generated for you, and you can use it to configure the corresponding nodes to interact with the MQ target server. In this way, a developer who understands the MQ interaction can discover the service and then just use the service description to configure the flows in an application. The discovery process can be described as a set of configuration steps that guide you through the process in a structured way.

Users enter connection details for the IT system repository to be discovered. After connecting, the system is queried for available technical assets. You can configure filters to omit irrelevant technical assets. The technical assets discovered may include interaction points (queues, programs, and so on) or data objects (tables, IDocs, and so on). Finally, a service description is created and all the related artifacts (WSDL and XSD files) are generated. Figure 1 shows the process for generating a service description using discovery:

Figure 1. Generating service descriptions using service discovery
Service discovery flow

An MQ service description is discovered from a live MQ system and consists of the MQ Service descriptor that encapsulates all OF the discovered configuration data and all OF the selected and generated artifacts. A WSDL file is generated that defines service interface and MQ binding details, XSD files, and message exchange format:

Figure 2. Service description
Service description

MQ service descriptions are created using the IBM Integration Bus Toolkit and configured using the New MQ Service wizard.

MQ service discovery: Connecting to MQ

After the MQ service description is created, the service file is displayed in the Toolkit Navigator under the Services folder. The Discovery Editor automatically opens for the newly created MQ service description. On the first discovery step, you will be able to view a detailed description of the MQ service discovery and usage, as shown in Figure 3 below. You can close and then continue the service discovery at any time by double-clicking the service element in the navigator.

Figure 3. MQ service discovery description
MQ service discovery description

The Connect step lets you specify the connection information for the MQ system that contains the resources that you want to use in your service:

Figure 4. MQ service connect
MQ service connect

Three connection options are supported by the MQ discovery process:

  • Local binding connection -- Uses only the queue manager name to connect to the queue manager on the MQ system installed in the same box where the discovery is performed.
  • Remote connection using CCDT file -- You can select the file path from the local file system using the Browse button. This file should contain all the connection information to establish a connection to MQ.
  • Remote connection properties -- Host name or IP address, port, and channel name.

After you have configured the connection details, you can test whether they let you connect to MQ by clicking Test Connection. When the connection to the target MQ system is established, the next step is enabled and you can proceed with selecting MQ resources for your service.

The Select Resources step automatically runs a query for the queues on the target queue manager using default filtering criteria. The message exchange format lets you select the service type that you want to create: One-Way or Request-Response. Filter text lets you narrow down the list of queues on the target MQ system. Include Resources flags let you view and select system and dynamic queues for your service:

Figure 5. Queues selected
Queues selected

The MQ Configuration step is an optional step for configuring MQ headers.

The final step is Service Configuration, which lets you describe the messages used by the service using an XML schema. At this discovery stage, you should select the input and output message descriptors. You select the XML structural feature using the selection dialog. The dialog contents is filtered based on the selected message configuration: XML schema elements or XML schema types. If the message structure cannot be described by XML, then you can just select string as a message type.

After you have selected and saved XML descriptors for the input and output messages, the service description WSDL file is generated and displayed as child element of the service node in the navigator. Click Open file for each message descriptor to open the corresponding XSD file to explore the structure of the XML schema structure, as shown in Figure 6 below. At this point, the service description discovery is complete.

Figure 6. Complete service description
Complete service description

The Summary page shows the overall structure of the discovered service interface as well as links to all the generated and referenced resources (WSDL and XSD). You can use the links to open the referenced files in the corresponding editor to explore the file structure, as shown in Figure 7:

Figure 7. Service summary
Service summary

Configuring integration flows using the MQ service

Once the MQ service is discovered, you can use it to configure message flows to interact with the MQ system using one of the following methods::

  • Drag and drop the MQ service file on the empty canvas of the Message Flow Editor, and all of the required nodes are created and configured using the service description.
  • Drag and drop the MQ Service file on the MQInput, MQOutput or MQGet nodes that exist on the message flow canvas. The target node is configured using the service description.
  • Configure the MQ node by selecting the service description queue on the node Properties view.

Implementing a service

If the selected service is of the Request-Response type, then the MQInput and MQOutput node are created and configured using the service response queue. The MQHeader node is created to set the proper response message MQ headers using the values from the service description and correlation ID to be copied from the original message ID. The MQInput node and MQHeader node are not connected, because this is where you may want to insert your own nodes that complete the implementation of the service. For a One-Way MQ service, only the MQInput node is created and configured, as shown in Figure 8:

Figure 8. Implement service flow nodes
Implemnet service flow nodes

MQ Input message parsing is configured based on the input message descriptor type selected during the discovery process.

Message parsing configuration rules
Input message descriptorNode message domainNode message
XSD type: stringBLOBN/A
XSD type: XSD Complex TypeXMLNSCN/A
XSD element: Global Named Element (DFDL)DFDLInput XSD element name
XSD element: Global Named Element (DFDL)DFDLInput XSD element name

If you select the second from the top Call a service option, then the MQoutput node is created and configured using the service request queue and queue manager. If the selected service is of the Request-Response type, then the MQGet node is created and configured using the service response queue and output message descriptor. The MQHeader node is created to set the proper response message MQ headers using the values from the service description, as shown below in Figure 9. For the One-way type MQ service, only an MQOutput node is created and configured.

Figure 9. Flow nodes calling a service
Flow nodes calling a service

When you drag and drop the MQ service on the existing flow, MQInput, MQOutput, or MQGet node, then this node is configured by the values from the service description. If the service is of the Request-Response type. then for the MQInput or MQOutput node you must select whether the service description request queue or response queue is to be used for node configuration.

Using the Integration Registry with an MQ service

Once the MQ service description is discovered, all files are stored in the Toolkit workspace. If you want to share the discovered MQ service with others who have access to the same Integration Registry, you can publish the MQ Service description to the selected Integration Registry. To open the Integration Registry view, use the Show View menu. The Integration Registry exists on every integration node, so you can publish the discovered MQ Service to the Integration Registry on any node that you are connected to. The Integration Registry view displays workload management policies and MQ service descriptions. Any published services can be downloaded by any user connected to the same IBM Integration Bus node, as shown in Figure 10 below, which enables easy collaboration between users.

Figure 10. Updated Integration Registry view
Updated Integration Registry view

Workload management (WLM)

IBM Integration Bus contains intelligent mechanisms to control the rate of processing within a message flow. This intelligence means that instead of tuning parameters such as number of threads or JVM heap size, you can simply specify the message flow rate that you want, and IBM Integration Bus will do the rest. Three WLM mechanisms are covered in this section:

  1. Decreasing the rate at which a message flow processes messages by sending a notification or delaying the processing of a message.
  2. Increasing the rate at which a message flow processes messages by allocating more threads.
  3. Monitoring a message flow and taking an action if it goes out of control to allow stuck message flows to be restarted.

All three of these mechanisms are implemented at the message flow level, with the message rate calculated at the point data enters the message flow through any Input node. All input nodes available in IBM Integration Bus can be used with these mechanisms. For example, if a message flow has an MQInput node that is processing messages at 5 messages/second and a HTTPInput node at 3 messages/second, then the total measured rate for the message flow is 8 messages/second (not the average of 4 messages/second.).

The main reason to decrease the message rate is to protect end systems from workload spikes that they cannot cope with in the short term. IBM Integration Bus has a notification threshold that you can set on a message flow that will cause an MQ message to be published whenever a threshold is crossed -- both when the rate exceeds the threshold, and when it drops below the threshold. The rate is calculated every 20 seconds and it does not noticeably increase the CPU cost of a running flow (similar to the cost of Statistics and Accounting). Figure 11 shows how to use this notification with the threshold set to 100 messages/second:

Figure 11. Notification threshold
Notification threshold

The notification threshold lets you receive the notification using MQ pub/sub and then take appropriate action. The notification follows the standard IBM Integration Bus monitoring event format, with custom elements for the rate at which the message flow is processing messages. It relies on you to take action to bring the rate back into bounds, and does not directly control the rate. Therefore the notification threshold does not directly protect a end application against high workloads. To actually limit the rate, IBM Integration Bus has a Maximum Rate parameter on the message flow. If the rate is too high, then the message flow is delayed until it decreases to the correct rate. Figure 12 shows a bursty sending application being smoothed out in a message flow using the Maximum Rate parameter:

Figure 12. Maximum rate
Maximum rate

The rate is constrained by delaying messages, and the total number of messages is the same in both graphs. The delay is calculated on a message-by-message basis: A timestamp is taken before receiving a message from the input node's transport (MQ or HTTP), and another timestamp is taken after all processing of the message is finished, including committing the transaction, and the message processing time is the difference between the two timestamps. If the difference is less than the value required to maintain the maximum rate, then the flow is delayed enough to smooth the rate back into range. The delay is done on a message-by-message basis rather than averaging over a period of time, and ensures that the end application workload never exceeds the maximum it can handle. Delaying messages is useful only if the average rate is below the maximum rate and the transport being used can handle the queued-up work that is not being processed due to the delays.

Increasing the rate at which a message flow can process messages is a more difficult problem than slowing things down. The message flow rate is constrained both by the rate at which the sending application sends data to the message flow, and by the available CPU and I/O resources on the machine. Message flows do scale very well, and you can add instances to enable more processing. You can also use the delaying mechanisms described above to control selected message flows and free up resources so that other flows can go faster. The threshold triggers a notification both when exceeded and when dropped below, so you can use it to send a notification when the rate is too slow.

Decreasing and increasing the message rate are important aspects of WLM. But it is also important to detect when a message flow becomes unresponsive or out of control, which can happen because of a loop in the flow or an interaction with an external system like a database. To handle these situations, you can set a message flow timeout, as shown in Figure 13, which specifies the maximum amount of time in seconds a message should take to be processed in a message flow. If the timeout is exceeded, then a notification message is published to MQ, and optionally, the execution group is restarted.

Figure 13. Processing timeout
Processing timeout

The timeout measurement is started when data is received in an input node, and an independent monitor keeps track of how long the flow has been processing and takes action when the timeout is exceeded. You can also use the mqsistopmsgflow command to stop a stuck message flow. The command has a -f option to enable you to force the stopping of a message flow by restarting the execution group.

You can set WLM properties on a message flow by using the BAR File Editor in the Toolkit, but the recommended way to set them is to define a separate policy document that is attached to the message flow.

Policies

IBM Integration Bus enables you to separate configuration data from message flows by using policies. They are similar to configurable services. The two main benefits of policies are that they are stored in the Integration Registry, and they can be edited as files, enabling source control mechanisms to be applied to them prior to promoting to a running server.

IBM Integration Bus provides the WLM policy type, which lets you configure message flow properties associated with WLM. A WLM policy can be attached to a message flow, which will then conform to the properties set on the policy. You can apply a single WLM policy to many message flows, and then updating that policy will affect all of those message flows. Even better, the update can be done at runtime and does not require the message flows to be restarted.

You can edit WLM policies using the IBM Integration Bus Web Admin, commands, the REST API, or the Integration Bus API. The Web Admin lets you create, read, update, and delete WLM Policies, as well as attach and detach those policies from message flows. Figure 28 shows the Policy Editor in the IBM Integration Bus Web Admin, and Figure 29 shows how to attach a WLM policy to a message flow:

Figure 14. WLM Policy Editor
WLM Policy Editor
Figure 15. Attaching a WLM policy
Attaching a WLM policy

You can also create, read, update, delete, attach, and detach WLM policies using the following commands:

  • mqsicreatepolicy
  • mqsireportpolicy
  • mqsiupdatepolicy
  • mqsideletepolicy
  • mqsiattachpolicy
  • mqsidetachpolicy

The IBM Integration Bus API includes new classes for handling policy objects as well as methods for attaching and detaching policies to and from message flows. Listing 1 shows how to retrieve WLM policies. For the complete code for bulk import and export of policies, see the Download section at the bottom of the article.

Listing 1. Sample WLM policy Message Broker API code
String policyType = "WorkloadManagement";
PolicyManagerProxy policyManagerProxy = brokerProxy.getPolicyManager(policyType);
if (policyManagerProxy != null) {
  String[] policyNamesByType = policyManagerProxy.getPolicyNames();
  if (policyNamesByType != null) {
    for (String policyName : policyNamesByType) {
      PolicyProxy policyProxy = policyManagerProxy.getPolicy(policyName);
      ...
    }
  }
}

The IBM Integration Bus REST API provides the following methods to create, read, update, and delete WLM policies:

  • /policy/WorkloadManagement
  • /policy/WorkloadManagement/{policyName}

Additionally, the IBM Integration Bus REST API provides methods to attach and detach WLM policies from message flows using the existing message flow methods.

Conclusion

This article discussed:

  • MQ and database discovery and how to use discovered services to generate message flows that call or implement those services
  • WLM concepts and new product features that let you control workload within message flows
  • Configuring WLM policies to manage the workload of message flows
  • How to use the IBM Integration Bus Integration Registry to store published service definitions and policies

Downloads

DescriptionNameSize
Code sample1MQServiceExamplePI.zip10 KB
Code sample2WLMPolicyImportExport.zip15 KB

Notes

  1. Archive with an application using the MQ services
  2. Sample code to bulk import and export policies

Resources

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into WebSphere on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=WebSphere
ArticleID=945485
ArticleTitle=Using workload management, service discovery, and the Integration Registry in IBM Integration Bus V9
publish-date=09182013