IBM BluemixDevelop in the cloud at the click of a button!

Enterprise event management framework, Part 1: Introducing the event framework for telecommunications

Business events are central to all operations within an organization, from handling service requests to managing accounting operations. Timely knowledge of business events allows you to respond quickly to customer actions. Understanding complex business event patterns permits the early detection of fraud, security intrusions, system failure conditions, and other situations detrimental to business operations. This two-part series presents and uses an enterprise event management framework for the telecommunications industry as a formal example.

Share:

Benjamin Lieberman , Ph.D., Principal Software Architect, BioLogic Software Consulting

author photoBen Lieberman serves as principal architect for BioLogic Software Consulting. He provides expertise in consulting and training on a variety of software development topics, including software architecture, requirements analysis, software analysis and design, configuration management, and development process improvement. Lieberman brings more than 10 years of software architecture and IT experience in various fields, including telecommunications, airline travel, e-commerce, government, financial services, and life sciences. His consulting services are based on the best practices of software development, with specialization in object-oriented architectures and distributed computing — in particular, Java™-based systems and distributed Web-site development (J2EE), XML/XSLT, Perl, and C++ based client-server systems. Lieberman has provided architectural services to corporate organizations, including EchoStar, Jones Cyber Solutions, Blueprint Technologies, Trip Network Inc., Galileo International; educational institutions, including Duke University and the University of Colorado; and governmental agencies, including the Mine Safety and Health Administration. He is also an accomplished professional writer with a book and numerous software-related articles to his credit. Lieberman holds a doctorate in biophysics and genetics from the University of Colorado, Health Sciences Center.



26 April 2011

Also available in

Event-driven business behavior

Businesses exist to provide products or services, or both, to customers. They do this by performing a wide variety of manual or automated business operations (processes) that directly provide the product or service, or support the delivery or performance of such. The trigger for these processes is a business event, where some set of conditions has been met, indicating that a specific business process is to be performed (see Resources for links to more information). For example, when a cellular customer wants to buy a new mobile phone, a customer purchase event occurs, indicating that the business must perform a series of operations to fulfill this request (such as updating inventory, arranging for billing, and provisioning of the desired services), as Figure 1 shows. (View a larger version of Figure 1.)

Figure 1. Event-driven business workflow
Image illustrating the subscriber ordering workflow

Note that these triggering events might initiate a cascade of other business events to complete the overall business process. In the preceding customer purchase event, the provisioning of services might require a number of internal systems to receive updates (such as a new service event) or even external partners who must respond to triggering business events with behaviors of their own. This complex business choreography must be successfully performed hundreds or even thousands of times each day for the business to be successful. Clearly, the ability to understand which business events have occurred (and when) is valuable in tracking and managing such complex business operations.


Responding to customer action

In addition to a focus on operations, businesses are placing increasingly high value on the idea of personal customer relationships (see Resources for links to more information). Rather than use broad categories of customer behavior (that is, demographics) to indicate which customer group prefers a product or service, organizations are analyzing individual customer behavior for targeted marketing. For example, many supermarket chains have customer loyalty programs that offer immediate discounts in return for capture of customer purchasing behavior. Later, this information is used to provide targeted coupons based on frequently purchased products. The ability to track individual customer-initiated actions (as a series of subsequent business events) provides a powerful mechanism for enhancing customer value through directed up-sale and cross-sale opportunities each time a customer interacts with the business.

Moreover, the ability to track these behaviors over time provides data required for predictive customer-retention actions. Most retention efforts are directed at reducing churn, or the likelihood that customers will defect to a competitor, but are typically reactive — that is, the retention action is taken only after a customer has indicated dissatisfaction with the business. There are often several critical warning signs (customer behaviors), though, that indicate that a customer is unhappy, such as frequent calls to customer service or the return of particular products. If a business is able to capture and track these events at an individual level, proactive steps can be taken to improve the customer experience and relationship with the company before it is irretrievably damaged.


Event-driven business intelligence

There is tremendous value in understanding the various business events that occur during normal (or abnormal) business operations. Capture and analysis of business event patterns (also known as complex event processing) enhance or enable a variety of business decisions, including the detection of fraudulent activities, auditing of financial drivers, and security intrusion detection. Application logging is a well-established means of gaining insight into the inner workings of internal business systems and has been used for many years to locate and eliminate system flaws. In recent years, log files have been applied to other areas of business intelligence to gain information on customer behaviors.

Unfortunately, file-based logging is insufficient for this purpose for several reasons. First, log entries are based on one-time system-level events and are not typically tied to the higher-level business drivers. Second, application developers are the main beneficiaries of logging efforts, so log entries vary in syntax and content and are often difficult to interpret, as these two examples show:

Example 1 (from Java application): 
08:50:49,664 FATAL LogClass:33 – Log Attempt Failed (btrack@metatelecommunications)

Example 2 (from web server):
April 3, 2010 (23:04:05) 45:T50 Log attempt failure – 772, 81gAt

Log information is needed at different operational levels (business, service, application, database, network, infrastructure) and for different purposes (business intelligence, operational support, security, auditing). As log entries are generated only within a specific system environment (such as a database, network device, or application), long-running business transactions — those of greatest interest to the business — are difficult to trace through multiple systems. Typically, no common mechanism exists to identify which log entries are tied to which others within a business transaction.

There are multiple issues with many current application-level logging mechanisms:

  • Applications, data stores, servers, and network elements all handle logging in a different way, using varying syntax, mechanisms, and trace levels (for example, fatal error, information, warning).
  • It is difficult, if not impossible, to tie together multiple levels of event generation (business. application, data store. or network) to determine causality between events (in other words, business event 1 caused application events A, B, C, and D).
  • Log information, which is driven more by development or operational needs rather than by business needs, might not include information critical to proper interpretation.
  • There is no concept of an event taxonomy (that is, a well-defined set of event description terms) to organize high- and low-level system events.
  • Significant additional processing is required to convert file-based logs into a searchable, understandable data store.
  • Interactions between multiple cooperating systems, as are required for complex transactions such as provisioning of a service, are not easily traced or connected because of the asynchronous nature of these operations.
  • There is no provision for complex event pattern matching, especially for real-time reaction to such important situations as customer-initiated events, fraud attempts, or security intrusions.

The enterprise event management framework

The purpose of the enterprise event management framework (EEMF) is to provide a lightweight mechanism for new and legacy applications within an organization to generate events and to be able to respond to events that other systems generate. In many respects, the EEMF is similar to a service-oriented architecture, but instead of defining an interface into a set of system services, the EEMF allows existing applications and services to log events to a commonly accessible location. Moreover, other systems that register for event notification (such as a complex event processor rule engine or near-real-time event business analytics) can be guaranteed notification of every event of interest.

The EEMF has five critical components:

  • Event taxonomy is a controlled vocabulary of business events that unambiguously identifies each event and the relationships to other events below and above it in a well-defined hierarchy.
  • Event identification is the globally unique identification of an event with respect to all other events, including subordinate events and superordinate events in an event tree.
  • Event structure is the core descriptive information about an event, including the origination of the event (system and user information) as well as the event-specific data used in the triggered business process.
  • Event transport is the mechanism by which an event is generated and responded to by formal, guaranteed delivery.
  • Event persistence is the long-term storage of event information for audit and business intelligence purposes.

Figure 2 illustrates these components.

Figure 2. Event management components
Illustration showing the five event management components

Event taxonomy

A controlled vocabulary and taxonomy

The term taxonomy represents a well-defined definition of related terms, typically organized into a tree-like structure. Taxonomies allow for "controlled vocabularies," which form the basis for a well-ordered referencing mechanism, especially for complex terms. For business events, the taxonomy is derived from the business operations, with additional specifying detail added as you move down the tree. As shown in Figure 3, the customer event tree is composed of several subnodes. These activities allows for all business processes that are customer initiated to be traced and tracked.

Event taxonomy provides a way to uniquely identify each type of event in an increasingly detailed manner (which is important during the discussion of the event data payload structure). Each term is uniquely identified by the event taxonomy ID based on the nature of the tree itself. For example, as Figure 3 shows, Customer is taxonomy ID A, Purchase is taxonomy ID A1, New Item is ID A13, and so forth. This approach of using one character position in the identifier per hierarchy level allows for insertion of new terms into the tree without requiring the renumbering of any of the other term identifiers.

Figure 3. Customer event hierarchy
Illustration of the customer event tree

Event identification

In addition to the event taxonomy, it is necessary to uniquely identify each event generated for two reasons: first, to allow events generated out of sequence to be placed in the proper order (for example, time-stamped), and second, to allow for the capture of event causality, or the ability to know which event caused another event to occur. This chain binds each event to both its parent and all the children events. Knowing any event in the tree, it is possible to reconstruct the entire tree using these unique identifiers (see Figure 4).

Figure 4. Purchase event tree
The customer event tree

To help trace parent-child relationships, the EEMF architecture introduces the concept of a propagating composite identifier (PCI), which originates at the top of the event tree and carries forward to all subsequent events. As shown in Listing 1, the PCI is created from a set of GUIDs (globally unique identifiers) that are passed forward to each subsequent event and that are appended with that event's GUID to create the composite identifier. In practical terms, this means that the event GUID becomes a required parameter in every event log call performed by applications or network elements participating in the event tree. For legacy systems, this may require wrapping application programming interface (API) calls in such a way as to capture the event before continuing the method invocation.

Listing 1. Propagating composite event identifier
Event #1: 
	Customer.Purchase.Change Composite Event ID =
				{4ef9ff68-d50d-4c0a-98cc-b60179ba238d}
Event #1.1: 
	Customer.Purchase.Change.Mobile_Number Composite Event ID =	
				{4ef9ff68-d50d-4c0a-98cc-b60179ba238d}::
				{83dfd56a-e973-4f74-9cf9-ae2a901f80d2}
Event #1.2: 
	Customer.Purchase.Download.Equipment Composite Event ID =	
				{4ef9ff68-d50d-4c0a-98cc-b60179ba238d}::
				{c5eb0984-fce4-45c7-8aa5-809da165ca15}
Event #1.2.1: 
	Customer.Purchase.Download.Equipment.Mobile Composite Event ID =
				{4ef9ff68-d50d-4c0a-98cc-b60179ba238d}::
				{c5eb0984-fce4-45c7-8aa5-809da165ca15}::
				{aed0dd16-6a1b-4bbd-90a4-a30f0c939bca}

When an event tree is completed, it can be easily reconstructed from any event in the tree by inspecting the composite key to obtain the parent key and then using the root-parent key (always the first key in the composite) to obtain and order all other related nodes.

To capture the structure of business transactions (which initiate each event tree), the event header of the payload contains an attribute for the transaction ID. However, the initiating application is responsible for including this value in the generation of the parent event. Events that are further down-chain have little use for this value and are not required to maintain or forward it to other events.

Event structure

Each event contains a set of information that is both global to all events (such as the event name, event GUID, and event key) and specific to that particular event. The basic structure of an event message is shown as an XML document in Figure 5 and Figure 6. The <event> tag represents the root element of the XML document and contains information global to all events, including the <event-context>. The event context allows each event to indicate where it was generated and who generated it and to provide details on the execution environment at the time of generation. The event context also contains information usable for security audits (such as a user ID), as shown in Figure 5.

Figure 5. Event XML schema structure
Image showing the event XML schema

The remainder of the event message is devoted to the event "payload" (Figure 6). The event payload contains information specific to each type of event, typically captured as XML attributes for each event level in the event taxonomy tree. For example, the <customer> tag contains information specific to the customer, such as first name, last name, and other identifying data. Each additional tag provides information to the overall event to give a complete picture of the full event tree.

Figure 6. Customer event XML structure
Image showing the customer XML schema

Notice also that the data provided at each level does not necessarily overlap as the event propagates from one system to the next. For example, the initiating application (that is, the customer care application) might know the top-level information about the customer and the purchase but not have information about lower-level events such as the directory number — MIN/MDN — or equipment number — MEID — that is assigned. Subsequent events might generate information that can be captured at that level by appending additional tags from the taxonomy and leaving higher-level tags empty.

Event transport

To use the defined event taxonomy, identification, and structure, a mechanism must be provided to application developers to easily create and post event messages. In the EEMF environment, this mechanism is provided through an enterprise service bus (ESB) and an enterprise messaging service (EMS). The ESB provides a set of services that allow event generators and subscribers to interact with the event message queue and topics. A summary of the event transport is provided in Figure 7. (View a larger version of Figure 7.)

Figure 7. The event transport service
Illustration of the event transport service

Event services are provided to allow the event generator application to use the GetEventGUID() method as the composite identifier for the event. Services also are provided to ValidateEvent() against the event hierarchy and PostEvent() for processing. Finally, parties can use the RegisterEventSubscriber() and DeregisterEventSubscriber() to register and de-register interest in particular events.

The events posted to the event message queue are immediately processed by the normalized message router to post the event message on a specific event topic. The event topics are organized based on the event taxonomy. For example, the first topic is the Customer event topic. Using the standard message-handling mechanisms, events submitted through the ESB to the EMS are configured for guaranteed delivery. Data quality for events is managed using the ValidateEvent() service to ensure that all event messages are well formed. The actual data contained in the event payload, however, is not validated at this time. Message filtering is handled at the subscriber–topic level. Events that fail validation are rejected back to the calling event generator as an error, and the error condition generates an exception event detailing the source of the erroneous event and the failed validations.

Event persistence

The events that are generated to the ESB/EMS need to be stored for long-term analysis and auditing. The event data mart repository has a defined retention window for how long events are retained. The event retention window is defined by the expressed needs for the retained event messages but should be at least 60 days (to provide time for analysis and business review). The length of time that events are stored depends primarily on the number of events generated and the amount of available storage space. It might be a reasonable compromise between on-demand data and storage costs to utilize first-tier storage for the first 60 days and then move data to a lower-cost second- or third-tier storage medium.

Based on the event payload definition provided, the estimates for data storage requirements are shown in Table 1.

Table 1. Event storage sizing chart
Message size (KB)Count/dayStore/day (GB)Store/month (GB)
5500,0002.575
10500,0005.0150
15500,0007.5225
5750,0003.75112.5
10750,0007.5225
15750,00011.25337.5
51,000,0005150
101,000,00010300
151,000,00015450

Conclusion

Business events represent a wealth of information not only about the performance of the business but also as a way to gain insight into the behaviors of customers and business partners. By using a well-defined event taxonomy and structure with a global mechanism for relating various events to one another and a transport mechanism available across the enterprise, the capture and analysis of all relevant business events are now possible. In the next article in this series, a prototype enterprise event management system will be developed to illustrate how such concepts work in practice.

Resources

Learn

  • Check out David Luckham's The Power of Events (Addison-Wesley, 2002) for a good introduction to the business value of event management and complex event processing.
  • For those with an interest in understanding the value and practice of developing business customer relationships, review Managing Customer Relationships by Don Peppers and Martha Rogers (John Wiley and Sons, 2004).
  • See to Marcus Wübben's doctoral thesis, Analytical CRM from Technische Universitat München (2008) for a good overview of customer relations management.
  • For a review of the power in real-time response to customer needs, read Kamakurq Wagner's chapter, "Cross-Selling — Offering the Right Product to the Right Customer at the Right Time,", in Profit Maximization Through Customer Relationship Marketing (L. Aksoy, T. Keiningham, D. Bejou, eds.; Haworth Press, 2008).
  • Industries library: See the developerWorks Industries library for technical articles and tips, tutorials, standards, and IBM Redbooks.
  • developerWorks technical events and webcasts: Stay current with technology in these sessions.
  • developerWorks on Twitter: Join today to follow developerWorks tweets.
  • developerWorks podcasts: Listen to interesting interviews and discussions for software developers.

Get products and technologies

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Business process management on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Business process management
ArticleID=650347
ArticleTitle=Enterprise event management framework, Part 1: Introducing the event framework for telecommunications
publish-date=04262011