A real-time enterprise is capable of recognizing and responding to actionable situations indicating threats or opportunities to the business as they occur. This series of articles describes how IBM uses a BPM framework to implement a real-time predictive solution that can detect and take action on problems in its New Product Announcement (NPA) business process. Part 1 described the process flow and challenges faced by NPA, which creates and publishes the IBM product and price catalog. NPA is a complex and highly distributed process; therefore, data quality is critical to customer satisfaction and sales. To address this need, IBM uses the Predictive Real-time Operational Business Intelligence Tool (PROBIT) framework. PROBIT is used to detect and resolve inaccuracies or publishing delays that impact customers and business partners.
Part 2 describes how PROBIT is implemented using WebSphere Business Events and WebSphere Business Monitor. The solution identifies complex business event patterns, which in turn provide automation, alerts, and a dashboard for business users of the system. The dashboard and alerts provide vital proactive information on real-time issues within the NPA process.
Problem Determination (PD) Tool, the solution for NPA, uses the following IBM products:
- WebSphere Business Events (hereafter referred to as Business Events) — provides the runtime event correlation engine and the design center environment for setting up the event processing rules, flows, and interactions.
- WebSphere Business Monitor (hereafter referred to as Business Monitor) — provides enriched dashboard capabilities and advanced alert functionality.
- WebSphere Message Broker (hereafter referred to as Message Broker) — provides content-based filtering, mediation, and routing to facilitate real-time integration with high-volume systems.
- Generalized Publish and Subscribe Services (GPASS) — extends Enterprise Service Bus (ESB) connectivity to simplify data synchronization.
This article contains with details on how the framework can be implemented to identify and act on patterns of complex business events. It covers installation of the framework, design considerations, an outline of the development process, and issues and recommendations.
The PD Tool solution
PD Tool is the internal IBM project name used for the implementation of the PROBIT event processing framework. Figure 1 shows the system context diagram for the PD Tool reference implementation. The left side of the diagram depicts the source systems where the events are formed and fed into the PD Tool Complex Business Event engine. Based on certain events or patterns of events occurring (or not occurring) as expected, the PD Tool alerts appropriate people of the situation via e-mail and dashboards. The alerts are based on roles and the nature of the situation detected. For example, if a product price is not released on time, according to a defined schedule, the tool alerts the âPricerâ (held in WWPRT) and a defined price coordinator (held in the PD Tool).
Figure 1. PD Tool system context diagram
The NPA system was developed to detect both new records related to NPA and updates to events already processed by Business Events and Business Monitor. The following products were deployed:
- Business Events V126.96.36.199: Complex business events
- Business Monitor V188.8.131.52: Dashboard with configurable views
- Message Broker V184.108.40.206: Filtering, mediation and routing
Figure 2 depicts the components in the reference implementation.
Figure 2. PD Tool implementation
The following patterns were implemented for the PD Tool solution. This article details the steps taken to produce some of the components used when implementing the Collect Pattern.
Collect Pattern: Used to collect and correlate data from events and external data sources. This pattern receives data from an event source, merges the event data with external data sources, stores pertinent event information in a database, sends data collection to Business Monitor, and initiates a scheduler event.
Update Pattern: Used to update data from prior events. This pattern receives data from an event, and retrieves and compares the data against stored event data. It sends appropriate changes in event data to Business Monitor and reschedules events if key date elements change.
Notify Pattern: Used to send alerts and notifications (can be time-dependent or periodically scheduled). This pattern receives data from a scheduler event and retrieves pertinent information from event data stored in a database. The combined information is used to identify whether an alert action (notification) is required.
Table 1 lists the event patterns processed by the PROBIT framework for the NPA business process.
Table 1. Event patterns in the PROBIT framework
|Collect Pattern||Detect NPA events|| Monitor NPA data — dashboard
Set timers for NPA business alerts
|Update Pattern||Detect NPA data updates (excluding Business Monitor keys)||Monitor NPA data — dashboard display|
|Move Pattern||Detect key changes (a key is identified by Business Monitor)||Monitor NPA data — dashboard display. If the Date of Announcement (DOA) changes, set new timers for NPA business alerts|
|Notify Pattern||Detect all products that become ready for pricing within 14 days* before DOA||Alert (e-mail) the pricing experts (Pricers) with list of products ready to be priced|
|Notify Pattern||Detect all products with a late status. Product status not promoted to FINAL in the EACM application within %14 days% before DOA||Display late status on Monitor dashboard|
|Notify Pattern||Detect all products that are not yet priced (late) in EACM within 8 days* before DOA||Display late status on Monitor dashboard|
|Notify Pattern||Detect all products whose price has not yet been released in EACM within 4 days* before DOA||Display late status on Monitor dashboard|
|Notify Pattern||Detect all products that are not yet released in the ePIMS application within 1 day* before DOA|| Display late status on Monitor
Send e-mail to GFS to release the Request for Announcements (RFAs)
|Notify Pattern||Detect all products whose Sales Status is not yet set to âSellableâ within 1 day* before DOA|| Display late status on Monitor
Send e-mail to GFS to update Sales Status
* All timer parameters are configurable from an external data source.
The following design considerations were taken into account for the solution:
- External configurable parameters. Actions and action properties (such as e-mail address and timer properties) are configurable from an external data source. There is support for additional instances of any one action with varying configuration properties.
- Event State. The solution uses an external data source to hold state between events. Event state is used to determine content change between NPA events.
- Scheduled events. An external configurable event scheduler is used to provide a flexible scheduling component. It is implemented with Message Broker but is not product-specific.
- ESB and Message Broker integration. Message Broker is used to filter the source file system and for mediation of events.
- Business Monitor integration. Monitor is used to expand the capabilities of the dashboard, reporting, and alert functionality.
Sample PD Tool usage pattern
The sample use case in Figure 3 shows how PD Tool processes the end-to-end (E2E) data quality file and sends an e-mail alert to a product Pricer notifying the person of several products that require pricing. On a subsequent date (calculated based on the event record and a timer parameter retrieved from a data source), another e-mail is sent to the same Pricer alerting the person that several products still require pricing and are now considered late. Event information is also sent to Business Monitor to be displayed on the dashboard.
Figure 3. Sample PD Tool usage pattern
Each step in the figure is summarized below. A link is included to more detailed information and illustrations.
- Identifies records required by the Business Events Collect Pattern (specific product types such as SEO, SVCPAC, BUNDLE are used).
- Mediates to event packet.
- Delivers event records to Business Events.
- Identifies whether an event is a new event and extracts the DOA.
- Retrieves configuration parameters from the EXPARMS database table (contains action timers and e-mail lists).
- Calculates timers based on DOA and timer parameters (typically timers are a certain number of days before DOA).
- Sends a scheduler event to the Message Broker scheduler (a scheduler event can contain one or more scheduler requests).
- Sends an announcement event to Business Monitor.
- Writes an event record to the NPASTORE database table.
- Updates the dashboard metrics and graphical views.
- Receives scheduler event from Business Events.
- Parses event and starts a timer for each scheduler request (time period is passed with each scheduler request).
- As each timer expires, the associated scheduled event is sent to the Business Events Notify Pattern.
- Receives a scheduled event from Message Broker.
- Retrieves event data from NPASTORE (this is specific to an individual notify implementation such as SalesStatusAlert).
- Sends an e-mail to the e-mail list (contained in scheduled event).
Step 1: E2E data quality file received by Message Broker
Figure 4 shows a sample E2E data quality file.
Figure 4. Sample E2E data quality file
Step 2: Message Broker consumes the file
- Filters the file by date and product type.
- Identifies and augments duplicate records.
- Identifies removed records (based on prior execution).
- Mediates Comma-separated values (CSV) event to the Event Connector Packet.
- Routes each record as a file to Business Events.
Step 3: Business Events consumes E2E event packets (created in step 2)
- Defines NPA touchpoint to receive events via a file connector.
Figure 5. Define Business Events touchpoints
- Retrieves EXPARMS (configurable properties) from a database table.
The table contains action-specific parameters such as e-mail and timer
Figure 6. Database of external configuration parameters and rules used for NPA solution
- Populates an intermediate object from the event and external data
Figure 7. Populate the intermediate objects in Business Events
- Builds and sends scheduler events to Message Broker.
Figure 8 - Schedule NPA events in Business Events
- Uses Business Events filters and interaction sets to determine if NPA
is an initial new event or an update event (New or Update NPA
- For an initial (new) event :
- Sends Message Broker announcement event to Business Monitor.
- Stores copy of event data in the NPASTORE database table.
- For an update event, identifies the change and:
- Sends Message Broker update event to Monitor.
- Updates event data in NPASTORE database table.
Figure 9. Sample NPA filters in Business Events for New and Update NPA events
- For an initial (new) event :
- Defines a Business Monitor touchpoint to send actions via a message
Figure 10. Business Monitor touchpoint defined in Business Events
Step 4: Dashboard (Business Monitor consumes events)
In this step, the process updates the dashboard metrics and graphical view as shown in Figure 11.
Figure 11. NPA process metrics displayed in Business Monitor dashboard
Step 5: Scheduler process (Message Broker consumes event packets created by Business Events)
- Creates a scheduled event for each event in the packet.
- When the scheduled date and time is reached :
- Creates an event connector packet.
- Routes the scheduled event to the Business Events input directory.
Figure 12. Message Broker scheduler process
Step 6: Notify Pattern (Business Events consumes the scheduler events generated by Message Broker)
In this step, the scheduled event packet is processed and filters are applied to the interaction set to determine which actions to take. E-mail actions are configurable and based on the content of the database source.
Figure 13. Sample Business Events alert action rules
Implementing the PROBIT architecture with IBM products
This section describes the steps used to implement the PROBIT architecture.
Installation and configuration
We used standard installation procedures to install all runtime components on a single AIX LPAR according to the product documentation. The following steps were taken:
- Install DB2® 9.5 (required level for Business Events) and create a database for the Business Events Repository.
- Install Business Monitor and WebSphere Application Server. We applied FixPak 220.127.116.11 which includes IFix PK66053 to prevent the connectors from hanging during startup.
- Install the Business Events run time. We chose the option to install on an existing WebSphere Application Server instance.
- Install and configure Message Broker including prerequisite software (WebSphere MQ).
We also installed WebSphere Business Event Design Data and Design tools on a local Windows XP machine (see Issues and recommendations for details on a timestamp issue).
This section discusses some of the design decisions related to integration with Message Broker and Business Monitor, and the tools used to design the Business Events system.
WebSphere Message Broker integration
Integration with Message Broker required the following design decisions:
- Filter and mediation. In the first release of the solution our input source for events was a spreadsheet with over 70,000 records. The file contained historical data referencing past product announcements as well as current and future announcement data. To reduce the workload on Business Events, Message Broker was implemented to filter records and mediate from a CSV record to an XML event connector format. Date elements were also mediated to YYYY-MM-DD format (which Business Events requires).
- Protocol switching. We chose to deliver the connector packet payload from Message Broker to Business Events by using a file node. This allowed us to avoid existing Business Events issues with Java Message Service (JMS). See Issues and recommendations for details.
- Scheduler. A scheduler service was created using Message Broker, although Business Events is also capable of creating delayed timer actions and events through the use of synthetic events. We chose to use an external service for scheduling, which provided an additional layer of decoupling for the solution.
For more detail on how to integrate Business Events and Message Broker see Resources.
WebSphere Business Events design tools
The following graphical user interfaces were used:
- Design Data is a Business Events component that supports the definition of the metadata layer required for Business Event Processing (BEP). Design Data is used to create the building blocks for the application. These include data sources, intermediate objects, touchpoints, events, and actions. See Terminology in WebSphere Business Events for definitions.
- Design is a Business Events component that supports the definition of filters and interaction sets used at run time by the Business Events runtime server to detect, correlate, and respond to simple and complex events.
WebSphere Business Monitor integration
Business Monitor is integrated with Business Events to provide advanced dashboard and notification capabilities. This solution uses Business Monitor to capture and display product information grouped by DOA and RFA. An instance view is used to show the NPA data in both tabular and graphical formats. Metrics are used to accumulate totals for specific events that have occurred. See Resources for detailed information on integrating Business Events and Business Monitor.
- Requirements for the data model. Prior to developing a Monitor context, we needed to model our event data. We required summary data containing metrics to count specific events based on the message content. We also needed to provide drill-down functionality to lower-level details of each record.
- Dashboard considerations. Figure 14 shows a tabular view that can be sorted by any column. The counts are collected from event contents received from Business Events. This view shows the total number of products per RFA along with corresponding accumulated totals of key metric indicators. The view also provides the capability to drill down to lower-level underlying details for an RFA.
Figure 14. Tabular view of NPA solution data in Business Monitor
Figure 15 shows the same data in chart format including:
- A graph of all DOAs
- A graph of individual DOAs with drill-down capability to show a graph view of the underlying RFA content
This view provides an easy method of spotting issues with current RFAs for a particular DOA or a historical view of past announcements.
In a future release this view will contain historical content for RFA metrics that were applied after missing key deadlines.
Figure 15. Graphical view of NPA solution data
Creating the components with WebSphere Business Events
This section describes the process used to create the components within Business Events. It is not intended to be a complete implementation guide but shows sample definitions used in the NPA solution.
Using the Design Data tool:
- Identify and create the touchpoints to each system (events and actions).
- Create an intermediate object that maps all the required fields of the NPA event to a common format to be shared between events and actions.
- a. Create an event that maps to the NPA event connector packet
b. Map event fields to the intermediate object.
- a. Create actions for the e-mail connector, JMS
connector (for Business
Monitor), and file connector (for logging and scheduler).
b. Map the intermediate object fields to the corresponding fields of each action object.
Using the Design tool:
- Create filters to check the Country code) and Prices Released fields.
- Create interaction sets to apply the filters and take action based on the events.
Step 1: Identify and create the touchpoints
This step determines which systems are required to interact with Business Events from either an event or action and defines touchpoints for each system. The NPA system interacts with the following:
- NPA system (for events)
- File system (for logs)
- E-mail system
- Business Monitor system (for integration with WebSphere Business Monitor)
To create a touchpoint, open Design Data and do the following:
- Create a new project and save it (note it is recommended that you save your project often).
- Right-click in the white space under Touchpoints and select Insert Touchpoint.
- Name the touchpoint (for this example, use NPA System).
Figure 16. Creating a touchpoint
Repeat this process adding all the required touchpoints.
Step 2: Create an intermediate object
This step creates the intermediate object that maps all the required fields of the NPA event to a common format to be shared between events and actions. The NPA system needs an intermediate object to be used as a common data-model mapping between each of our events and actions. In this example we are using a single intermediate object to map between our objects but it is also possible to use multiple smaller intermediate objects.
To create an intermediate object, open the Design Data and do the following:
- Right-click in the white space under Intermediate Objects and select Insert Touchpoint.
- Select Blank...
- Name the touchpoint (for this example, NPA_Object).
Figure 17. Inserting an intermediate object
Then create the fields that compose the intermediate object.
- Select the new intermediate object just created.
- Right-click and select Insert Intermediate Object Fieldâ¦Insert
- Enter the details for the field (Name and Data Type).
- Repeat this process until you have added all the required fields.
Figure 18. Inserting an intermediate object field
Step 3a: Create an event
This step shows how to create the event and event object that maps to the NPA event connector packet message format. It also outlines the process for mapping the event object fields to the intermediate object.
To create the event, do the following:
- Right-click the NPA System touchpoint (created in the previous step).
- Select Insert Event then select Normal...
- Name the event NPA Record.
Figure 19. Creating an event
To configure the event:
- Right-click the NPA Record event and select Event Properties.
- Select the Connection tab and choose the connector required for the event (in our case File Connector is required).
- Choose the appropriate Format from the drop-down menu (in our case Connector Packet).
- Click Configure and enter the File Directory and Pattern options.
- Configure the appropriate polling interval.
Once complete the configuration should look similar to Figure 20.
Figure 20. Configuring an event
Add an event object:
- Right-click the NPA Record event.
- Select Insert Event Object then select Blank...
- Name the event object NPA Record.
Figure 21. Adding an event object
Finally complete the event object definition by adding the fields that make up the event:
- Right-click the event object (NPA Record).
- Select Insert Event Object Field (or use the Insert key).
- Enter the name and type.
- Repeat this process until all the required fields have been created.
Step 3b: Map the event object to the intermediate object
Step 4a: Create actions
In this step we create actions and action objects for the e-mail connector, JMS connector (for Business Monitor), and file connector. We then map fields from the intermediate object to the action objects.
This step creates the action and action object, which maps the intermediate object to Business Monitor. To create the action:
- Right-click on the Business Monitor system touchpoint.
- Select Insert Action then select Normal...
- Name the event Business Monitor Announcement.
Figure 22. Creating the action
Configure the action:
- Right-click the Business Monitor Announcement event.
- Select Action Properties.
- Select the Connection tab.
- Choose the connector required for the action (in our case Message Queue Connection is required)
- Choose the appropriate Message Type from the drop-down menu (in our case CBE).
- Populate the rest of the options for CEI (see Figure 23 and Figure 24).
- Click Configure and enter the JMS configuration.
Figure 23. Configuring the connection
Figure 24. Configuring the JMS provider
Add an action object:
- Right-click the Business Monitor Announcement action.
- Select Insert Action Object then select Blank...
- Name the action object Business Monitor Announcement.
Figure 25. Adding an action object
Finally complete the action object definition by adding the fields that make up the action.
- Right-click the action object (Business Monitor Announcement).
- Select Insert Action Object Field (or use the Insert key).
- Enter the name and type.
- Repeat this process until all the required fields have been created.
Step 4b: Map intermediate object to the action
After creating the action object, the intermediate object fields can be mapped to the corresponding fields in the action object. Just as we did when we mapped the event object, we also need to map the action object fields. This time we map in the opposite direction: from the intermediate object to the action object fields. The easiest way to accomplish this is to drag the field from the intermediate object and drop it over the field in event object that it should map to.
Step 5: Create filters to check the Country code and Prices Released fields
Once all the events, actions and intermediate objects have been created and configured using the Design Data tool, the Design tool is used to build filters and interactions sets.
To create a new filter, open the Design tool and do the following:
- Open the same project that was created in the Design Data tool (it is recommended that you close the Design Data tool when editing the project in the Design tool).
- Click Filters to create a new filter component.
- Rename the filter.
- Configure the filter (in our example we will create a filter to check
Country for a value of
- Click on the first Operand and select the intermediate object field we want to check (Country in this case).
- Click the Operator and select Equals Ignoring Case.
- Click on the second Operand, select Enter Text, then enter
USas the value.
Figure 26. Creating a new filter
Repeat this process to create all filters required; see the product documentation for all the possible filters that can be used.
Step 6: Create interaction sets to apply the filters and take action based on the events
Interaction sets use events and filters to determine an outcome to take through actions. This example demonstrates how to invoke a synthetic event.
- Click on Interaction Sets to create a new one.
- Rename the interaction set.
- Click the In Response To field and choose an event to be processed (NPA Record, in this case).
- Right-click the top green bar of the new interaction set and select Define context relationship. Choose an element from the intermediate object (SEO in our case). A context identifies a key that ties events together.
- Click the Action field (next to Always) and choose one of the available actions. In the case where we want to fire a synthetic action, we choose Synthetic Event.
- Right-click the grey area that surrounds the action component and select Add Filter. Then choose from the available filters.
After following these steps the result should look similar to Figure 27.
Figure 27. The filter for Monitor new US NPA announcement
This completes this interaction set; repeat these steps to create additional interaction sets. In our implementation we have interaction sets that fire actions for e-mails, logging, and sending events to Business Monitor.
Issues and recommendations
The following issues were discovered when implementing the PROBIT solution with Business Events:
- Choice of JMS Provider. Although the preferred integration method between Message Broker and Business Events is to use JMS, V6.1 of Business Events currently only supports a single JMS provider. Because the solution uses JMS to integrate with Business Monitor, we had to use file integration to Message Broker. A request has been opened with Business Events development to support multiple JMS providers.
- Timestamps. The timestamp for both the run time and design environment must be identical when deploying code to the repository. The Business Events run time fails to start when it contains a project deployed from a different time or timezone.
This article series describes how IBM used a BPM framework to implement a real-time predictive solution that can detect and resolve problems in its product catalog operation. Part 1 presents the Predictive Real-time Operational Business Intelligence Tool (PROBIT) architecture. Part 2 contains details on how PROBIT was implemented using WebSphere Business Events and WebSphere Business Monitor. It describes the process flow and provides guidelines on how to design the components of the Business Events system. The reference architecture and implementation can be reused to support any end-to-end business process that requires you to correlate interrelated events and detect exception situations.
- GPASS: A generalized publish and subscribe solution using WS-Notification standards, Part 1 : An introduction (Akram Bou-Ghannam and Matt Roberts, developerWorks 2008): Generalized Publish and Subscribe Services (GPASS), built on the implementation of Web Services-Notification in WebSphere Application Server V6.1, extends ESB connectivity to simplify data synchronization, thus helping you dramatically cut the cost of information distribution in your enterprise. Part 1 of this series describes GPASS architecture and implementation and highlights SOA case studies from IBM that use the GPASS framework.
- GPASS: A generalized publish and subscribe solution using WS-Notification standards, Part 2: Building SOA applications with reusable integration services (Akram Bou-Ghannam, Matt Roberts, and Allandel Manipon, developerWorks 2008): Part 2 of this series on GPASS provides an overview of the WSN specification and its implementation in WebSphere Application Server V6.1, and describes how it has been used in a GPASS implementation to solve the SAP Ledger scenario introduced in Part 1.
- Read about the Gartner Event Processing Summit.
- How Event Processing Improves Business Decision Making (K. Mani Chandy and W. Roy Schulte, Complex Event Processing, 2007): A good overview of CEP and its positive impact on business.
- OASIS WS-Notification Technical Committee home page: Get more information on WS-Notification, and download the specification.
- Improving information access and reuse with SOA, Part 1: An architecture to help your enterprise become information-centric in an SOA world (Akram Bou-Ghannam and Andrew Hoppe, developerWorks 2008): This article describes an enterprise information strategy and architectural framework to maximize the value and accessibility of information in an enterprise, and to help your enterprise become information-centric in an SOA world.
- Business Activity Monitoring with WebSphere Business Monitor V6.1: Through step-by-step hands-on examples, this IBM Redbook explores how you can use WebSphere Business Monitor V6.1 to track business events from a variety of applications. We discuss combining WebSphere Business Monitor with process-based applications running in WebSphere Process Server, WebSphere MQ Workflow, and FileNet P8 to achieve business activity monitoring.
- Business Activity Monitoring: The definition in Wikipedia.
- Using IBM WebSphere Message Broker as an ESB with WebSphere Process Server: This IBM Redbook is for architects who are planning an SOA solution and application designers who are implementing an SOA solution with WebSphere Process Server and WebSphere Message Broker. In this book, we highlight the ESB capabilities of WebSphere Message Broker and explain how you can leverage them with WebSphere Process Server. In addition, we discuss interoperability and provide examples to illustrate the integration of the two products.
- WebSphere Business Events: Get complete product information, including features and benefits and a demo. Read the IBM whitepaper "Empowering the business to sense and respond: Delivering Business Event Processing with IBM WebSphere Business Events."
- WebSphere Business Events Getting Started Guide and Tutorial Files: The Getting Started Guide contains instructions about how to build and run the tutorial application.
- Business Event Processing with WebSphere Business Events, Part 3: Using WebSphere Business Events with WebSphere Message Broker (Xiaoming Zhang, developerWorks 2008).
- Business event processing with WebSphere Business Events, Part 1: An introduction (Peter Crocker, Doina Klinger, Latha Sivakumar, developerWorks 2008).
- Business Event Processing with WebSphere Business Events, Part 5: Integrating Business Events with WebSphere Business Monitor (Latha Sivakumar, developerWorks 2008).
- The Power of Events (David Luckham, Addison-Wesley Professional, 2002): Refer to this book for more information on event hierarchies and complex events.
- developerWorks BPM zone: Get the latest technical resources on IBM BPM solutions, including downloads, demos, articles, tutorials, events, webcasts, and more.