Enable the real-time enterprise with business event processing, Part 1: A real-world enterprise scenario from IBM shows you how

This article uses a real-world scenario to illustrate the need for a framework that supports business event processing. This article is the first in a two-part series that describes how such a framework is used to help identify errors before they occur. This content is part of the IBM Business Process Management Journal.

Akram Bou-Ghannam, Ph.D. (akram@us.ibm.com), Executive IT Architect (Senior Certified), IBM

Akram Bou-Ghannam photoDr. Bou-Ghannam is an Executive IT Architect (Senior Certified) leading efforts to transform IBM into a globally integrated on-demand enterprise through SOA and dynamically reconfigurable services and architectures. His expertise in complex event processing and intelligent agents is helping IBM to realize the smart and agile enterprise – an enterprise that is capable of real-time discovery and response to actionable business situations. His SOA enterprise information architecture extends SOA into the data layer through the provision of information services, and helps achieve the enterprise vision of becoming information-centric in an SOA world. Dr. Bou-Ghannam is an IBM Master Inventor, and an accomplished author with multiple internal and external publications to his name.

You can reach Dr. Bou-Ghannam at akram@us.ibm.com.



Paul Faulkner (pfaulkn@us.ibm.com), Certified IT Specialist, IBM

Paul Faulkner photoPaul Faulkner is a Certified IT Specialist with IBM Software Services for WebSphere, specializing in system integration. Paul has designed and implemented complex integration patterns, including policy-driven Enterprise Service Bus (ESB) and solutions involving complex business event processing. Paul has over 20 years experience in IT with the majority of time spent developing middleware solutions. He has presented at multiple conferences. You can contact Paul at pfaulkn@us.ibm.com.



13 February 2009 (First published 04 December 2008)

Also available in Chinese Spanish

Introduction

In this article, we'll use a real-world scenario from IBM, the New Product Announcement end-to-end business process, to illustrate how a Business event processing (BEP) framework can help you realize a real-time enterprise. The real-time enterprise is capable of recognizing situations as they arise, of anticipating and responding to threats before they occur, and of discovering and capitalizing on opportunities. The core of the framework is WebSphere Business Events (hereafter called WBE), which provides the event processing engine, and WebSphere Business Monitor (hereafter called Monitor), which provides the rich dashboard. Event transformations and connectivity functions are provided by WebSphere Message Broker (Message Broker), and the Generalized Publish and Subscribe Services (GPASS).

In Part 1 of this series, we'll describe how the BEP framework is used to help identify errors in the IBM product and price catalog before they occur and before they can impact customers. With the help of the BEP framework, IBM can ensure that catalog data set-up is complete, correct, and timely in support of key business processes such as New Product Announcement (NPA). This implementation has helped to resolve data quality problems in IBM to improve credibility, customer satisfaction, and revenue. The framework and all the associated reference implementations have been documented as the basis for an engagement model allowing the reuse of the framework in other service engagements to support any end-to-end business process.

In the following sections, we'll introduce the NPA scenario and give an overview of BEP and the need for real-time operational business intelligence. We'll then present the architecture of our BEP framework, the Predictive Real-time Operational Business Intelligence Tool (PROBIT), which embodies the vision. Next, we'll describe how to realize this architecture using various IBM products. Finally, we'll present the NPA use case scenario from the IBM enterprise, and describe a reference implementation using PROBIT to manage the NPA process.


The NPA scenario

This section introduces the New Product Announcement (NPA) scenario from the IBM enterprise, and explains why a solution approach using BEP is needed. When implementing the NPA scenario, a clear requirement was an engine or a framework that supports BEP for enabling the real-time enterprise. The solution was to build a BEP framework, called Predictive Real-time Operational Business Intelligence Tool, or PROBIT, based on the following IBM products:

  • WebSphere Business Events is the run-time event correlation engine and the integrated development environment for setting up the event processing rules, flows, and interactions.
  • WebSphere Business Monitor provides enriched dashboard capabilities and advanced alert functionality.
  • [optional] WebSphere Message Broker provides content-based filtering, mediation and routing to facilitate real-time integration with high-volume systems.

In addition, PROBIT utilizes the Generalized Publish and Subscribe Services (GPASS) ([1], [2]) to distribute events to event consumers.


The need for real-time operational business intelligence in the IBM enterprise

IBM delivers product and price data (catalogs) to its customers and business partners on a regular basis. When something goes wrong, such as a product is missing from the catalog, or a product in the catalog has an incorrect price, there is usually a customer impact, which in turn has a negative impact on IBM’s credibility and can directly impact IBM’s revenue. IBM wants to be able to identify patterns that lead to catalog errors before they impact customers.

The current process of creating, updating, and publishing the product and price information (catalog) is complex and highly distributed across applications in the enterprise. Enterprise business processes such as the NPA process or the Price Action process drive and affect the catalog development and distribution operations across the entire enterprise. In order to identify the patterns that may lead to catalog errors (and quickly and appropriately act upon the detection of these patterns), we need to capture, evaluate, and correlate multiple events from the various systems and organizations that are involved across the enterprise, as shown in Figure 1. So, the action we take to prevent catalog errors from happening is not triggered by a single event, but by a complex composition of events, happening at different times, and within different contexts. The type of processing needed for this problem is called business event processing, or BEP. Figure 1 illustrates a BEP component processing various NPA events.

Figure 1. BEP applied to the NPA process
BEP in NPA process

An overview of BEP

Event processing is the ability to detect and respond to events (or activities) occurring across the enterprise. Event processing, in general, adds a dynamic dimension to an application by enabling insight that facilitates decision-making based on observations about things that happen in the enterprise. It can help in identifying trends, threats, opportunities, and business situations that need action. Event processing is not a new concept. For the last forty years, organizations have been utilizing a form of simple event processing to detect and respond to a single source, or homogeneous, event type. An example is: If Event A occurs then do X. A simple event process might specify that, if a shipment received event occurs, the quantity should be added to an inventory database.

In recent years, emerging technologies like Complex Event Processing (CEP) and Event Stream Processing (ESP) have begun to tackle problems of greater complexity that traditional event processing could not handle. For instance, with CEP you can "analyze, correlate and summarize low-level events into higher-level events suitable for notifying people in human terms or for triggering automated processes." [3] CEP and ESP employ techniques such as detecting complex patterns across many events, using rule processing algorithms for event correlation and abstraction, using event hierarchies and relationships between events. Analysis of causality, membership, timing, and event-driven processes are core capabilities of these technologies. Business Event Processing (BEP), is the next generation of event processing, extending the capabilities and tools of technologies like CEP and ESP to the business user to define and detect situations in business context for rapid response to opportunities and threats.

So what are the characteristics of BEP systems, or to put in another way, what are the application characteristics best suited for BEP? As opportunities and threats appear at unpredictable times, event-driven systems must respond to events at times that are externally determined. These systems act when the world changes and not only according to pre-planned schedules. Typically, BEP is applicable in scenarios where many components need to come together in real time in order for a task to be completed. BEP is best suited if you if answer yes to at least one of these application requirements questions:

  1. Do I need to detect event patterns occurring across disparate sources over varying timeframes?
  2. Do my applications require support for unpredictable sequences or timing of event occurrences?
  3. Do I need dynamic resolution of response processing and exception handling?
  4. Are my processing rules frequently changing?
  5. Do I want my business users (business analysts and so on) to generate and maintain the processing rules?

Predictive Real-time Business Intelligence Tool (PROBIT): The architecture and vision of a situational awareness framework

An event processing system receives events from an event producer, processes these events, possibly creates new events as a result of the processing, and sends the processed events to event consumers, as shown in Figure 2. The event processing functions include validation, enrichment, routing, transformation, orchestration, and pattern detection.

Applications, files, databases, feeds, people, sensor data, and more form the various information and event producers and consumers in an enterprise environment. An event is an abstraction that represents the fact that something happened or is happening, such as a stock trade, a customer order, an address change, and so on. A computer application creates an event object (a computer record) to signal or report the event. A notification is a computer message (for example, an XML message) that consists of an event object.

Figure 2. BEP components
BEP components

Figure 3 shows the high-level architecture of an intelligent system capable of providing situational awareness for predictive, real-time operational business intelligence. This high-level architecture forms the basis for the architecture of the PROBIT tool. This architecture is divided into layers of abstraction starting with the external environment layer, the sensing and actuating layer, the connectivity layer, and the cognitive layer.

The external environment consists of all the applications and systems, including people, in the enterprise. These applications constitute the operational business environment in an enterprise, and are the sources for observations about things that happen in the enterprise. Monitoring, capturing, and processing these observations can help in identifying trends, threats, opportunities, and business situations in the enterprise that need reaction. The external environment thus includes not only the various event producers, but the various event consumers, as well.

The Sensing and Actuating layer consists of components that sense and act upon the external environment. Sensors are components that detect and capture the events that happen in the environment. For example, an Address Change event can signify the fact that John Doe changed his address at 8 AM today. These events maybe logged in a Changed Address table in a database. A sensor component could be a piece of code, such as a script, a stored procedure, and so on, that monitors the Changed Address table and generates an event object (in the form of an XML message, for example) every time a new record is created. The sensor component could also filter the observed data, then translate and transform it into a different format. For example, the sensor component can publish the Address Change event as a WS-Notification [4] standard Notify message, where the event object XML is contained in the payload of the Notify message. At a minimum, a sensor component must perform the monitoring and detecting functions to detect that the event happened and to create an event object to signal the event. In the architecture depicted in Figure 3, the sensor component must also publish a notification message that includes the actual event object to the broker component in the connectivity layer. A sensor component could also include filtering, translation and transformation capabilities.

The Actuator component acts upon the environment. The action typically changes the state of the environment, such as changing the state of a database application by setting a flag indicating that the address is now considered valid and ready to go through the address standardization process. The action might also involve alerting people to certain threats in the environment by sending email, phone or pager notifications. The input to the actuator is an event object. Event consumers in the environment may not be able to consume the event object in its current form. For example, the consumer may be a SAP application that can only consume SAP-type objects (like SAP IDOC) via a specific SAP interface or API (like the SAP RFC or BAPI interface). In this case, the actuator must act as an SAP adapter that not only establishes connectivity with the SAP system (for example, via an RFC client), but also transforms the event object to the SAP object format. The actuator thus extends the specific SAP interface, and translates the event object to the SAP object format. The actuator in this case is acting as a proxy or agent on behalf of the event consumer to consume, transform, and send the event object to the consumer.

Figure 3. PROBIT logical architecture
PROBIT logical architecture

The connectivity layer mainly highlights a broker component within the context of the PROBIT architecture. The main role of the broker is to route events from event producers to event consumers. Note that the BEP engine in the cognitive layer is also an event consumer. The broker component functionality can be achieved via a publish and subscribe framework, such as the Generalized Publish and Subscribe Services (GPASS) [1,2]. In general, the connectivity layer represents the Enterprise Service Bus (ESB) architectural construct for SOA. You can think of it as a layer that provides asynchronous and synchronous communications across the enterprise and the extended enterprise, using techniques like messaging, method calls, service integration and FTP. It virtualizes the services by decoupling the point-to-point connections from the interfaces themselves. The service interfaces are put into a third-party broker, which helps you manage the interfaces better, and enables faster and more flexible coupling and decoupling of applications. Because you can find all of the applications and the interfaces, you can more easily reuse them.

The cognitive layer houses the engines that host and process the business rules, representing the business domain knowledge. A BEP component is one such engine that can help in detecting important patterns in the environment, or enterprise. The BEP component is an event consumer that receives and deals with multiple events, often of disparate types coming from disparate event sources. BEP employs various techniques to find patterns in business event data, and to help enable operational business intelligence in the enterprise, as described earlier. The BEP component is also an event producer that generates events (typically higher level events that identify the patterns detected). Other event consumers that subscribe to these events receive them via the broker component through their corresponding proxy (adapter, agent, or actuator) component.


The PROBIT architecture in an SOA world

Figure 4 represents the SOA view of the PROBIT architecture. This architecture merges event processing services with the SOA information access and distribution architecture [5]. In this architecture, information services constitute most of the sensing and actuating for the event processing component. For example, data event monitoring services perform the "sensor" function by detecting changes in the information sources and creating and publishing events. The semantic and logical services can help translate or transform event objects, if needed.

The connectivity and interoperability services shown in Figure 4 constitute the ESB and perform the event broker functions illustrated in Figure 3. This functionality can be implemented using IBM ESB products like Message Broker or WebSphere ESB.

Figure 4. PROBIT SOA architecture
PROBIT SOA architecture

The BEP component is represented as part of the event processing services. The BEP services consume and process events, and produce new higher level events. This functionality can be implemented using WBE. However, the dashboard functionality supported by WBE is very limited in its capabilities. For advanced dashboard and business activity monitoring (shown in Figure 4 above the Event Processing Services component), you can use Monitor.

The information services shown in Figure 4 can be developed using the IBM Information Server and its tools. Some of the semantic and logical data transformation services can be implemented using WebSphere Transformation Extender. Other services requiring integration with enterprise applications such as SAP or CRM Siebel can be easily developed using the IBM WebSphere Adapter products.

The next section describes in more detail the IBM products used to implement the PROBIT architecture.


Implementing the PROBIT architecture with IBM products

The approach we took to implement the PROBIT framework solution architecture uses various IBM products including WBE, Monitor, and Message Broker. To fully implement the various components of the PROBIT architecture as shown in Figures 3 and 4, you can use the following IBM products:

PROBIT component IBM products to implement the component
Event Processing Services and BEP component WBE can be used as the correlation engine, or run-time, for identifying patterns of interaction among multiple disparate events. The interaction sets, or event correlation rules, can be created with the WBE build-time environment. WBE supports complex business processes by providing the ability to easily handle the context of a business process activity--that is, its presence or absence, timing, sequence, and relationship to other activities. Business Events provides a basis for full support of BEP.
Business Activity Monitoring – Dashboard alerts and notifications Monitor is comprehensive business activity monitoring (BAM) software that provides users with a real-time end-to-end view of business processes and operations. Monitor provides customizable business dashboards that calculate and display key performance indicators (KPIs) and metrics derived from business processes, business activity data, and business events from a wide range of information sources. Business users can view these KPIs, metrics, and alerts through various means including lightweight Web interfaces, Smartphones such as Blackberry devices and iPhones, corporate portals, and on desktops. With Monitor, business users can set up and manage alerts with minimal involvement from IT.

Monitor is not just a dashboard visualization for events processed by WBE, but also provides event processing capabilities that enable insight into business activity and can detect business situations relative to how the business is performing.

Connectivity / Broker component Message Broker is a powerful information broker that allows both business data and information, in the form of messages, to flow between disparate applications and across multiple hardware and software platforms. Rules can be applied to the data that is flowing through the message broker in order to route, store, retrieve, and transform the information. Message Broker provides universal connectivity, including Web services, and any-to-any data transformation. Monitor is one of the IBM's ESB product offerings.

The use case: Managing the NPA business process in IBM

IBM delivers product and price data (catalogs) to its customers and business partners on a regular basis. The current process of creating, updating, and publishing the product and price information is complex and highly distributed across applications in the enterprise. The current process relies on the synchronization of multiple data points in the information flow in order for the correct data to be provided. The information delivered is collected from multiple disparate sources distributed across the enterprise. Often the same information is entered by different applications at different times (multiple data entry) causing inconsistencies in the data. In order for the process to succeed, the right information must be provided at the right time in the information flow. Otherwise, the timely information managed by the process won't make it to the customers and business partners. Customers and partners initiate a problem report when they don't receive the needed information. For example, a business partner may complain that product X is missing from the EDI Parts Price file.

Problem determination is difficult and time-consuming. It crosses multiple teams and skill sets. When something goes wrong, tracing back through the flow to determine when and where a failure occurred is complex and time-consuming. In addition, to satisfy the requirements of new projects and initiatives, it is difficult, costly, and time-consuming to modify the data flows of the current process and environment and to ensure that the data flow is set-up correctly and properly synchronized.

When you modify the data flows as part of an enterprise transformation initiative, the process of ensuring that the data is correctly set up and synchronized is complex and can impact your ability to meet cost and schedules, especially for System Integration Test (SIT). It would be advantageous to have a tool not only to help in problem determination efforts when problems occur, but also to help identify data errors before they impact customers. Such a tool would have a positive impact on the business. Also, ensuring that the data set-up is complete and correct prior to testing would avoid test failures due to data errors and thus reduce testing costs.

In addition to capturing and monitoring current events, this solution provides a means to report statistical data by capturing historical views of past NPAs.


Reference implementation:

Part 2 of this series will provide in-depth details of the solution implemented to manage the IBM NPA. This reference implementation will give you insight into capturing events from multiple systems and correlating them to produce complex event scenarios. Part 2 will also discuss the integration of WBE with Message Broker to provide content-based filtering and with Monitor to enable advanced dashboard and alert mechanisms.

Figure 5 shows the components implemented in the reference implementation.

Figure 5. Reference implementation
Reference implementation

Summary

In this article, we used a real-world scenario from IBM to highlight the need for a framework that supports BEP to enable the real-time enterprise. We gave you an overview of BEP and the need for real-time operational business intelligence, and described the architecture of the Predictive Real-time Operational Business Intelligence Tool (PROBIT). PROBIT enables a real-time enterprise, which is capable of recognizing situations as they arise in real-time, anticipating and responding to threats before they occur, and discovering and capitalizing on opportunities. At the core of this framework is WebSphere Business Events, which provides the event processing engine, and WebSphere Business Monitor, which provides a rich dashboard.

In Part 2 in this series, you'll learn how to implement the event processing solution to manage the IBM NPA. You'll also learn how WebSphere Business Events is integrated with WebSphere Message Broker to provide content-based filtering and with WebSphere Business Monitor to enable advanced dashboard and alert mechanisms.

Resources

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into WebSphere on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=WebSphere, SOA and web services
ArticleID=355629
ArticleTitle=Enable the real-time enterprise with business event processing, Part 1: A real-world enterprise scenario from IBM shows you how
publish-date=02132009