Introduction

A real-time enterprise is capable of recognizing and responding to actionable situations indicating threats or opportunities to the business as they occur. This series of articles describes how IBM uses a BPM framework to implement a real-time predictive solution that can detect and take action on problems in its New Product Announcement (NPA) business process. Part 1 described the process flow and challenges faced by NPA, which creates and publishes the IBM product and price catalog. NPA is a complex and highly distributed process; therefore, data quality is critical to customer satisfaction and sales. To address this need, IBM uses the Predictive Real-time Operational Business Intelligence Tool (PROBIT) framework. PROBIT is used to detect and resolve inaccuracies or publishing delays that impact customers and business partners.

Part 2 describes how PROBIT is implemented using WebSphere Business Events and WebSphere Business Monitor. The solution identifies complex business event patterns, which in turn provide automation, alerts, and a dashboard for business users of the system. The dashboard and alerts provide vital proactive information on real-time issues within the NPA process.

Problem Determination (PD) Tool, the solution for NPA, uses the following IBM products:

• WebSphere Business Events (hereafter referred to as Business Events) — provides the runtime event correlation engine and the design center environment for setting up the event processing rules, flows, and interactions.
• WebSphere Business Monitor (hereafter referred to as Business Monitor) — provides enriched dashboard capabilities and advanced alert functionality.
• WebSphere Message Broker (hereafter referred to as Message Broker) — provides content-based filtering, mediation, and routing to facilitate real-time integration with high-volume systems.
• Generalized Publish and Subscribe Services (GPASS) — extends Enterprise Service Bus (ESB) connectivity to simplify data synchronization.

This article contains with details on how the framework can be implemented to identify and act on patterns of complex business events. It covers installation of the framework, design considerations, an outline of the development process, and issues and recommendations.

The PD Tool solution

PD Tool is the internal IBM project name used for the implementation of the PROBIT event processing framework. Figure 1 shows the system context diagram for the PD Tool reference implementation. The left side of the diagram depicts the source systems where the events are formed and fed into the PD Tool Complex Business Event engine. Based on certain events or patterns of events occurring (or not occurring) as expected, the PD Tool alerts appropriate people of the situation via e-mail and dashboards. The alerts are based on roles and the nature of the situation detected. For example, if a product price is not released on time, according to a defined schedule, the tool alerts the “Pricer” (held in WWPRT) and a defined price coordinator (held in the PD Tool).

The NPA system was developed to detect both new records related to NPA and updates to events already processed by Business Events and Business Monitor. The following products were deployed:

• Business Events V6.1.0.1: Complex business events
• Business Monitor V6.1.2.0: Dashboard with configurable views
• Message Broker V6.1.0.2: Filtering, mediation and routing

Figure 2 depicts the components in the reference implementation.

Event patterns

The following patterns were implemented for the PD Tool solution. This article details the steps taken to produce some of the components used when implementing the Collect Pattern.

Collect Pattern: Used to collect and correlate data from events and external data sources. This pattern receives data from an event source, merges the event data with external data sources, stores pertinent event information in a database, sends data collection to Business Monitor, and initiates a scheduler event.

Update Pattern: Used to update data from prior events. This pattern receives data from an event, and retrieves and compares the data against stored event data. It sends appropriate changes in event data to Business Monitor and reschedules events if key date elements change.

Notify Pattern: Used to send alerts and notifications (can be time-dependent or periodically scheduled). This pattern receives data from a scheduler event and retrieves pertinent information from event data stored in a database. The combined information is used to identify whether an alert action (notification) is required.

Table 1 lists the event patterns processed by the PROBIT framework for the NPA business process.

* All timer parameters are configurable from an external data source.

The following design considerations were taken into account for the solution:

• External configurable parameters. Actions and action properties (such as e-mail address and timer properties) are configurable from an external data source. There is support for additional instances of any one action with varying configuration properties.
• Event State. The solution uses an external data source to hold state between events. Event state is used to determine content change between NPA events.
• Scheduled events. An external configurable event scheduler is used to provide a flexible scheduling component. It is implemented with Message Broker but is not product-specific.
• ESB and Message Broker integration. Message Broker is used to filter the source file system and for mediation of events.
• Business Monitor integration. Monitor is used to expand the capabilities of the dashboard, reporting, and alert functionality.

Sample PD Tool usage pattern

The sample use case in Figure 3 shows how PD Tool processes the end-to-end (E2E) data quality file and sends an e-mail alert to a product Pricer notifying the person of several products that require pricing. On a subsequent date (calculated based on the event record and a timer parameter retrieved from a data source), another e-mail is sent to the same Pricer alerting the person that several products still require pricing and are now considered late. Event information is also sent to Business Monitor to be displayed on the dashboard.

Each step in the figure is summarized below. A link is included to more detailed information and illustrations.

Step 1 – The E2E file is sent to Message Broker.

Step 2 – The Message Broker Parse/Filter process.

• Identifies records required by the Business Events Collect Pattern (specific product types such as SEO, SVCPAC, BUNDLE are used).
• Mediates to event packet.
• Delivers event records to Business Events.

Step 3 – The Business Events Collect Pattern

• Identifies whether an event is a new event and extracts the DOA.
• Retrieves configuration parameters from the EXPARMS database table (contains action timers and e-mail lists).
• Calculates timers based on DOA and timer parameters (typically timers are a certain number of days before DOA).
• Sends a scheduler event to the Message Broker scheduler (a scheduler event can contain one or more scheduler requests).
• Sends an announcement event to Business Monitor.
• Writes an event record to the NPASTORE database table.

Step 4 - Dashboard

• Updates the dashboard metrics and graphical views.

Step 5 – Scheduler process

• Receives scheduler event from Business Events.
• Parses event and starts a timer for each scheduler request (time period is passed with each scheduler request).
• As each timer expires, the associated scheduled event is sent to the Business Events Notify Pattern.

Step 6 – Business Events Notify Pattern

• Receives a scheduled event from Message Broker.
• Retrieves event data from NPASTORE (this is specific to an individual notify implementation such as SalesStatusAlert).
• Sends an e-mail to the e-mail list (contained in scheduled event).

Step 1: E2E data quality file received by Message Broker

Figure 4 shows a sample E2E data quality file.

Step 2: Message Broker consumes the file

1. Filters the file by date and product type.
2. Identifies and augments duplicate records.
3. Identifies removed records (based on prior execution).
4. Mediates Comma-separated values (CSV) event to the Event Connector Packet.
5. Routes each record as a file to Business Events.

Step 3: Business Events consumes E2E event packets (created in step 2)

1. Defines NPA touchpoint to receive events via a file connector.
   
   Figure 5. Define Business Events touchpoints
   
   
   
2. Retrieves EXPARMS (configurable properties) from a database table. The table contains action-specific parameters such as e-mail and timer delays.
   
   Figure 6. Database of external configuration parameters and rules used for NPA solution
   
   
   
3. Populates an intermediate object from the event and external data source.
   
   Figure 7. Populate the intermediate objects in Business Events
   
   
   
4. Builds and sends scheduler events to Message Broker.
   
   Figure 8 - Schedule NPA events in Business Events
   
   
   
5. Uses Business Events filters and interaction sets to determine if NPA is an initial new event or an update event (New or Update NPA event).
   • For an initial (new) event :
     • Sends Message Broker announcement event to Business Monitor.
     • Stores copy of event data in the NPASTORE database table.
   • For an update event, identifies the change and:
     • Sends Message Broker update event to Monitor.
     • Updates event data in NPASTORE database table.
   
   
   Figure 9. Sample NPA filters in Business Events for New and Update NPA events
   
   
   
6. Defines a Business Monitor touchpoint to send actions via a message queue connector.
   
   Figure 10. Business Monitor touchpoint defined in Business Events
   
   
   

Step 4: Dashboard (Business Monitor consumes events)

In this step, the process updates the dashboard metrics and graphical view as shown in Figure 11.

Step 5: Scheduler process (Message Broker consumes event packets created by Business Events)

1. Creates a scheduled event for each event in the packet.
2. When the scheduled date and time is reached :
   • Creates an event connector packet.
   • Routes the scheduled event to the Business Events input directory.

Step 6: Notify Pattern (Business Events consumes the scheduler events generated by Message Broker)

In this step, the scheduled event packet is processed and filters are applied to the interaction set to determine which actions to take. E-mail actions are configurable and based on the content of the database source.

Implementing the PROBIT architecture with IBM products

This section describes the steps used to implement the PROBIT architecture.

Installation and configuration

We used standard installation procedures to install all runtime components on a single AIX LPAR according to the product documentation. The following steps were taken:

1. Install DB2® 9.5 (required level for Business Events) and create a database for the Business Events Repository.
2. Install Business Monitor and WebSphere Application Server. We applied FixPak 6.1.0.19 which includes IFix PK66053 to prevent the connectors from hanging during startup.
3. Install the Business Events run time. We chose the option to install on an existing WebSphere Application Server instance.
4. Install and configure Message Broker including prerequisite software (WebSphere MQ).

We also installed WebSphere Business Event Design Data and Design tools on a local Windows XP machine (see Issues and recommendations for details on a timestamp issue).

Design considerations

This section discusses some of the design decisions related to integration with Message Broker and Business Monitor, and the tools used to design the Business Events system.

WebSphere Message Broker integration

Integration with Message Broker required the following design decisions:

• Filter and mediation. In the first release of the solution our input source for events was a spreadsheet with over 70,000 records. The file contained historical data referencing past product announcements as well as current and future announcement data. To reduce the workload on Business Events, Message Broker was implemented to filter records and mediate from a CSV record to an XML event connector format. Date elements were also mediated to YYYY-MM-DD format (which Business Events requires).
• Protocol switching. We chose to deliver the connector packet payload from Message Broker to Business Events by using a file node. This allowed us to avoid existing Business Events issues with Java Message Service (JMS). See Issues and recommendations for details.
• Scheduler. A scheduler service was created using Message Broker, although Business Events is also capable of creating delayed timer actions and events through the use of synthetic events. We chose to use an external service for scheduling, which provided an additional layer of decoupling for the solution.

For more detail on how to integrate Business Events and Message Broker see Resources.

WebSphere Business Events design tools

The following graphical user interfaces were used:

• Design Data is a Business Events component that supports the definition of the metadata layer required for Business Event Processing (BEP). Design Data is used to create the building blocks for the application. These include data sources, intermediate objects, touchpoints, events, and actions. See Terminology in WebSphere Business Events for definitions.
• Design is a Business Events component that supports the definition of filters and interaction sets used at run time by the Business Events runtime server to detect, correlate, and respond to simple and complex events.

WebSphere Business Monitor integration

Business Monitor is integrated with Business Events to provide advanced dashboard and notification capabilities. This solution uses Business Monitor to capture and display product information grouped by DOA and RFA. An instance view is used to show the NPA data in both tabular and graphical formats. Metrics are used to accumulate totals for specific events that have occurred. See Resources for detailed information on integrating Business Events and Business Monitor.

• Requirements for the data model. Prior to developing a Monitor context, we needed to model our event data. We required summary data containing metrics to count specific events based on the message content. We also needed to provide drill-down functionality to lower-level details of each record.
• Dashboard considerations. Figure 14 shows a tabular view that can be sorted by any column. The counts are collected from event contents received from Business Events. This view shows the total number of products per RFA along with corresponding accumulated totals of key metric indicators. The view also provides the capability to drill down to lower-level underlying details for an RFA.

Figure 15 shows the same data in chart format including:

• A graph of all DOAs
• A graph of individual DOAs with drill-down capability to show a graph view of the underlying RFA content

This view provides an easy method of spotting issues with current RFAs for a particular DOA or a historical view of past announcements.

In a future release this view will contain historical content for RFA metrics that were applied after missing key deadlines.

Creating the components with WebSphere Business Events

This section describes the process used to create the components within Business Events. It is not intended to be a complete implementation guide but shows sample definitions used in the NPA solution.

Using the Design Data tool:

1. Identify and create the touchpoints to each system (events and actions).
2. Create an intermediate object that maps all the required fields of the NPA event to a common format to be shared between events and actions.
3. a. Create an event that maps to the NPA event connector packet message format.

   b. Map event fields to the intermediate object.

4. a. Create actions for the e-mail connector, JMS connector (for Business Monitor), and file connector (for logging and scheduler).

   b. Map the intermediate object fields to the corresponding fields of each action object.

Using the Design tool:

1. Create filters to check the Country code) and Prices Released fields.
2. Create interaction sets to apply the filters and take action based on the events.

Step 1: Identify and create the touchpoints

This step determines which systems are required to interact with Business Events from either an event or action and defines touchpoints for each system. The NPA system interacts with the following:

• NPA system (for events)
• File system (for logs)
• E-mail system
• Business Monitor system (for integration with WebSphere Business Monitor)

To create a touchpoint, open Design Data and do the following:

1. Create a new project and save it (note it is recommended that you save your project often).
2. Right-click in the white space under Touchpoints and select Insert Touchpoint.
3. Name the touchpoint (for this example, use NPA System).

Repeat this process adding all the required touchpoints.

Step 2: Create an intermediate object

This step creates the intermediate object that maps all the required fields of the NPA event to a common format to be shared between events and actions. The NPA system needs an intermediate object to be used as a common data-model mapping between each of our events and actions. In this example we are using a single intermediate object to map between our objects but it is also possible to use multiple smaller intermediate objects.

To create an intermediate object, open the Design Data and do the following:

1. Right-click in the white space under Intermediate Objects and select Insert Touchpoint.
2. Select Blank...
3. Name the touchpoint (for this example, NPA_Object).

Then create the fields that compose the intermediate object.

1. Select the new intermediate object just created.
2. Right-click and select Insert Intermediate Object Field…Insert
3. Enter the details for the field (Name and Data Type).
4. Repeat this process until you have added all the required fields.

Step 3a: Create an event

This step shows how to create the event and event object that maps to the NPA event connector packet message format. It also outlines the process for mapping the event object fields to the intermediate object.

To create the event, do the following:

1. Right-click the NPA System touchpoint (created in the previous step).
2. Select Insert Event then select Normal...
3. Name the event NPA Record.

To configure the event:

1. Right-click the NPA Record event and select Event Properties.
2. Select the Connection tab and choose the connector required for the event (in our case File Connector is required).
3. Choose the appropriate Format from the drop-down menu (in our case Connector Packet).
4. Click Configure and enter the File Directory and Pattern options.
5. Configure the appropriate polling interval.

Once complete the configuration should look similar to Figure 20.

Add an event object:

1. Right-click the NPA Record event.
2. Select Insert Event Object then select Blank...
3. Name the event object NPA Record.

Finally complete the event object definition by adding the fields that make up the event:

1. Right-click the event object (NPA Record).
2. Select Insert Event Object Field (or use the Insert key).
3. Enter the name and type.
4. Repeat this process until all the required fields have been created.

Step 3b: Map the event object to the intermediate object

After creating the event object, the fields can be mapped to the corresponding fields in the intermediate object. The easiest way to do this is to drag the field from the event object definition and drop it over the fields in the intermediate object that it should map to. Intermediate objects can also be resolved from other sources including external data sources, JavaScript and hard-coded constants.

Step 4a: Create actions

In this step we create actions and action objects for the e-mail connector, JMS connector (for Business Monitor), and file connector. We then map fields from the intermediate object to the action objects.

This step creates the action and action object, which maps the intermediate object to Business Monitor. To create the action:

1. Right-click on the Business Monitor system touchpoint.
2. Select Insert Action then select Normal...
3. Name the event Business Monitor Announcement.

Configure the action:

1. Right-click the Business Monitor Announcement event.
2. Select Action Properties.
3. Select the Connection tab.
4. Choose the connector required for the action (in our case Message Queue Connection is required)
5. Choose the appropriate Message Type from the drop-down menu (in our case CBE).
6. Populate the rest of the options for CEI (see Figure 23 and Figure 24).
7. Click Configure and enter the JMS configuration.

Add an action object:

1. Right-click the Business Monitor Announcement action.
2. Select Insert Action Object then select Blank...
3. Name the action object Business Monitor Announcement.

Finally complete the action object definition by adding the fields that make up the action.

1. Right-click the action object (Business Monitor Announcement).
2. Select Insert Action Object Field (or use the Insert key).
3. Enter the name and type.
4. Repeat this process until all the required fields have been created.

Step 4b: Map intermediate object to the action

After creating the action object, the intermediate object fields can be mapped to the corresponding fields in the action object. Just as we did when we mapped the event object, we also need to map the action object fields. This time we map in the opposite direction: from the intermediate object to the action object fields. The easiest way to accomplish this is to drag the field from the intermediate object and drop it over the field in event object that it should map to.

Step 5: Create filters to check the Country code and Prices Released fields

Once all the events, actions and intermediate objects have been created and configured using the Design Data tool, the Design tool is used to build filters and interactions sets.

To create a new filter, open the Design tool and do the following:

1. Open the same project that was created in the Design Data tool (it is recommended that you close the Design Data tool when editing the project in the Design tool).
2. Click Filters to create a new filter component.
3. Rename the filter.
4. Configure the filter (in our example we will create a filter to check Country for a value of US.
5. Click on the first Operand and select the intermediate object field we want to check (Country in this case).
6. Click the Operator and select Equals Ignoring Case.
7. Click on the second Operand, select Enter Text, then enter US as the value.

Repeat this process to create all filters required; see the product documentation for all the possible filters that can be used.

Step 6: Create interaction sets to apply the filters and take action based on the events

Interaction sets use events and filters to determine an outcome to take through actions. This example demonstrates how to invoke a synthetic event.

1. Click on Interaction Sets to create a new one.
2. Rename the interaction set.
3. Click the In Response To field and choose an event to be processed (NPA Record, in this case).
4. Right-click the top green bar of the new interaction set and select Define context relationship. Choose an element from the intermediate object (SEO in our case). A context identifies a key that ties events together.
5. Click the Action field (next to Always) and choose one of the available actions. In the case where we want to fire a synthetic action, we choose Synthetic Event.
6. Right-click the grey area that surrounds the action component and select Add Filter. Then choose from the available filters.

After following these steps the result should look similar to Figure 27.

This completes this interaction set; repeat these steps to create additional interaction sets. In our implementation we have interaction sets that fire actions for e-mails, logging, and sending events to Business Monitor.

Issues and recommendations

The following issues were discovered when implementing the PROBIT solution with Business Events:

• Choice of JMS Provider. Although the preferred integration method between Message Broker and Business Events is to use JMS, V6.1 of Business Events currently only supports a single JMS provider. Because the solution uses JMS to integrate with Business Monitor, we had to use file integration to Message Broker. A request has been opened with Business Events development to support multiple JMS providers.
• Timestamps. The timestamp for both the run time and design environment must be identical when deploying code to the repository. The Business Events run time fails to start when it contains a project deployed from a different time or timezone.

Summary

This article series describes how IBM used a BPM framework to implement a real-time predictive solution that can detect and resolve problems in its product catalog operation. Part 1 presents the Predictive Real-time Operational Business Intelligence Tool (PROBIT) architecture. Part 2 contains details on how PROBIT was implemented using WebSphere Business Events and WebSphere Business Monitor. It describes the process flow and provides guidelines on how to design the components of the Business Events system. The reference architecture and implementation can be reused to support any end-to-end business process that requires you to correlate interrelated events and detect exception situations.

Resources

• GPASS: A generalized publish and subscribe solution using WS-Notification standards, Part 1 : An introduction (Akram Bou-Ghannam and Matt Roberts, developerWorks 2008): Generalized Publish and Subscribe Services (GPASS), built on the implementation of Web Services-Notification in WebSphere Application Server V6.1, extends ESB connectivity to simplify data synchronization, thus helping you dramatically cut the cost of information distribution in your enterprise. Part 1 of this series describes GPASS architecture and implementation and highlights SOA case studies from IBM that use the GPASS framework.
  
  
• GPASS: A generalized publish and subscribe solution using WS-Notification standards, Part 2: Building SOA applications with reusable integration services (Akram Bou-Ghannam, Matt Roberts, and Allandel Manipon, developerWorks 2008): Part 2 of this series on GPASS provides an overview of the WSN specification and its implementation in WebSphere Application Server V6.1, and describes how it has been used in a GPASS implementation to solve the SAP Ledger scenario introduced in Part 1.
  
  
• Read about the Gartner Event Processing Summit.
  
  
• How Event Processing Improves Business Decision Making (K. Mani Chandy and W. Roy Schulte, Complex Event Processing, 2007): A good overview of CEP and its positive impact on business.
  
  
• OASIS WS-Notification Technical Committee home page: Get more information on WS-Notification, and download the specification.
  
  
• Improving information access and reuse with SOA, Part 1: An architecture to help your enterprise become information-centric in an SOA world (Akram Bou-Ghannam and Andrew Hoppe, developerWorks 2008): This article describes an enterprise information strategy and architectural framework to maximize the value and accessibility of information in an enterprise, and to help your enterprise become information-centric in an SOA world.
  
  
• Business Activity Monitoring with WebSphere Business Monitor V6.1: Through step-by-step hands-on examples, this IBM Redbook explores how you can use WebSphere Business Monitor V6.1 to track business events from a variety of applications. We discuss combining WebSphere Business Monitor with process-based applications running in WebSphere Process Server, WebSphere MQ Workflow, and FileNet P8 to achieve business activity monitoring.
  
  
• Business Activity Monitoring: The definition in Wikipedia.
  
  
• Using IBM WebSphere Message Broker as an ESB with WebSphere Process Server: This IBM Redbook is for architects who are planning an SOA solution and application designers who are implementing an SOA solution with WebSphere Process Server and WebSphere Message Broker. In this book, we highlight the ESB capabilities of WebSphere Message Broker and explain how you can leverage them with WebSphere Process Server. In addition, we discuss interoperability and provide examples to illustrate the integration of the two products.
  
  
• WebSphere Business Events: Get complete product information, including features and benefits and a demo. Read the IBM whitepaper "Empowering the business to sense and respond: Delivering Business Event Processing with IBM WebSphere Business Events."
  
  
• WebSphere Business Events Getting Started Guide and Tutorial Files: The Getting Started Guide contains instructions about how to build and run the tutorial application.
  
  
• Business Event Processing with WebSphere Business Events, Part 3: Using WebSphere Business Events with WebSphere Message Broker (Xiaoming Zhang, developerWorks 2008).
  
  
• Business event processing with WebSphere Business Events, Part 1: An introduction (Peter Crocker, Doina Klinger, Latha Sivakumar, developerWorks 2008).
  
  
• Business Event Processing with WebSphere Business Events, Part 5: Integrating Business Events with WebSphere Business Monitor (Latha Sivakumar, developerWorks 2008).
  
  
• The Power of Events (David Luckham, Addison-Wesley Professional, 2002): Refer to this book for more information on event hierarchies and complex events.
  
  
• developerWorks BPM zone: Get the latest technical resources on IBM BPM solutions, including downloads, demos, articles, tutorials, events, webcasts, and more.
  
  

About the authors

Paul Faulkner is a Certified IT Specialist with IBM Software Services for WebSphere, specializing in system integration. Paul has designed and implemented complex integration patterns, including policy-driven Enterprise Service Bus (ESB) and solutions involving complex business event processing. Paul has over 20 years experience in IT with the majority of time spent developing middleware solutions. He has presented at multiple conferences.

You can reach Paul at pfaulkn@us.ibm.com.

Dr. Bou-Ghannam is an Executive IT Architect (Senior Certified) leading efforts to transform IBM into a globally integrated on-demand enterprise through SOA and dynamically reconfigurable services and architectures. His expertise in complex event processing and intelligent agents is helping IBM to realize the smart and agile enterprise – an enterprise that is capable of real-time discovery and response to actionable business situations. His SOA enterprise information architecture extends SOA into the data layer through the provision of information services, and helps achieve the enterprise vision of becoming information-centric in an SOA world. Dr. Bou-Ghannam is an IBM Master Inventor, and an accomplished author with multiple internal and external publications to his name.

You can reach Dr. Bou-Ghannam at akram@us.ibm.com.

Comments (Undergoing maintenance)

Back to top

Trademarks  |  My developerWorks terms and conditions

Table of contents

• Introduction
• The PD Tool solution
• Sample PD Tool usage pattern
• Implementing the PROBIT architecture with IBM products
• Creating the components with WebSphere Business Events
• Issues and recommendations
• Summary
• Resources
• About the authors
• Comments

Next steps from IBM

• Try: WebSphere Business Events (in the SOA Sandbox: Process)
• Demo: Reduce business risk with WebSphere Business Events
• Webcast: Develop innovative applications efficiently using the right application foundation in today's economy
• Briefing: WebSphere flexible business processes
• Buy: WebSphere Business Events (in the SOA Sandbox: Process)

My developerWorks community

Dig deeper into WebSphere on developerWorks

Rate a product. Write a review.

Write a review for Rational products today!

Special offers