IBM Intelligent Operations Center key performance indicators (KPIs), Part 1: Defining a low-level KPI

The new IBM Intelligent Operations Center is designed to help cities manage and improve their operations. It provides a key performance indicator (KPI) subsystem that allows cities to monitor and assess performance of city services, programs, and other resources. This series of articles shows how KPIs are modeled, implemented, and tested. This first article focuses on the development of a low level KPI. To help illustrate the concepts, we will examine how a sample KPI is modeled.

Share:

Allen Smith (allens@us.ibm.com ), Senior Certified IT Specialist, IBM

Allen Smith photoAllen Smith is a Senior Certified IT Specialist a member of the IBM Industry Solutions Development group in Research Triangle Park, North Carolina. He develops new content that extends the Industry Solutions offerings.



12 July 2011

Also available in Portuguese

IBM Intelligent Operations Center

The IBM Intelligent Operations Center is a new product from IBM designed to help cities manage and improve their operations. It is part of the IBM Government Industry Framework, which provides a software platform and roadmap for implementing IBM Smarter Cities solutions. The Intelligent Operations Center is based on several IBM software technologies designed to process event information in a cohesive manner, and present that information to city personnel in a centralized, real-time, collaborative environment. Views tailored to city executives allow key personnel with cross-organization responsibility to monitor, manage, and respond to status changes in relation to key areas of organizational performance. Views tailored to operators and managers provide detailed event information that allow them to interrogate and manage events, and take action as needed.


Common Alerting Protocol

The key underpinning of the Intelligent Operations Center is the event management subsystem. The event management subsystem receives, manages, stores, and disseminates events to the appropriate Intelligent Operations Center components for processing. Using a common event format ensures effective, seamless event processing. Produced by OASIS/ITU-T 9, the Common Alerting Protocol (CAP) provides the common event format (see Resources). CAP specification defines an open, non-proprietary XML-based data format for exchanging alerts and notifications. Although initially intended for broadcasting public alerts and warnings, the format is flexible enough to apply to all event type messaging between similar and disparate systems. The Intelligent Operations Center can also process Non-CAP events by normalizing them to the CAP event structure before injecting them into the system for processing.

A CAP message consists of an <alert> segment, which may contain one or more <info> segments, each of which may include one or more <area> and/or <resource> segments.

  • The <alert> segment provides basic information about the current message such as its purpose, its sender, and its status.
  • The <info> segment provides the event details. It describes the event, indicates when it occurred, its severity, and may give instructions for an appropriate response by message recipients and other information about the event.
  • The <area> segment describes a geographic area to which the <info> segment applies.
  • The <resource> segment provides a reference to additional information related to the <info> segment such as an image file.

Key performance indicators

Key performance indicators (KPIs) are quantifiable measurements employed by organizations to monitor and assess performance. While KPIs differ from one organization to the next, their purpose remains the same. They help reduce complex organizational performance information into a consumable format that allows organizations to more easily assess performance by comparing KPIs to the organization's stated objectives.

The Intelligent Operations Center provides a KPI subsystem, tooling, and visual components allowing organizations to define and monitor KPIs meaningful to their goals and objectives. A set of sample KPIs are installed with the product that demonstrate the functionality and monitoring capability of the Intelligent Operations Center KPI subsystem and visual components. The source for the sample KPIs is provided allowing organizations to view or modify the KPIs using the KPI tooling.


IBM Intelligent Operations Center KPI engine

The Intelligent Operations Center KPI subsystem uses IBM Business Monitor as the KPI processing engine. The KPIs themselves are defined in monitor model applications. A monitor model is an extensible markup language (XML) document that describes a set of metrics and KPIs, their relationship to incoming events, conditions warranting action, and outbound events that report these conditions. For the Intelligent Operations Center, CAP messages represent the incoming events. CAP messages flow to the Intelligent Operations Center and based on the message content, a message may be deemed a KPI event and forwarded to the Business Monitor server for processing. Metrics hold information used as input to KPI calculations. For the Intelligent Operations Center, metrics hold information passed in CAP messages. The conditions warranting actions and the outbound events reporting the conditions are the mechanisms used by the Intelligent Operations Center to receive asynchronous notifications of KPI changes.


KPI tooling

Monitor models and KPIs are developed and modified using the IBM Business Monitor development toolkit. The toolkit provides wizards, editors, and libraries for developing monitor models and a test environment for testing the models. The IBM Business Monitor development toolkit can be installed into Rational Application Developer or WebSphere Integration Developer. The Intelligent Operations Center models were developed using Rational Application Developer and version 7.0.0.3 of the development toolkit. The IBM Business Monitor test environment provides a fully functional Business Monitor server, Business Space, and the Business Monitor widgets. You can use the Business Space and the Business Monitor widgets to display KPIs outside an Intelligent Operations Center environment for testing purposes.


IBM Intelligent Operations Center sample KPIs

The Intelligent Operations Center includes three sample monitor models. The icoc sample public safety monitor model provides a set of Public Safety KPIs, the icoc sample transportation monitor model provides a set of Transportation KPIs and the icoc sample water monitor model provides a set of Water KPIs. IBM provides the source for the Intelligent Operations Center sample monitor models with the Intelligent Operations Center product. The source for the monitor models is contained in an archive file that can be imported into Rational Application Developer or WebSphere Integration Developer with the IBM Business Monitor toolkit installed. Once imported, you can change, add, or delete KPI definitions, and re-generate and re-deploy the applications. The sample models illustrate the patterns and conventions used for defining Intelligent Operations Center KPIs.

The Intelligent Operations Center sample KPIs are defined as nested KPIs, meaning there is a parent-child relationship between KPIs. Figure 1 shows the parent-child relationships between the Police Department KPIs. In this example, Police Department is the parent of Crime Response Time, which is the parent of Crime Response Time Precinct One and Crime Response Time Precinct Two. The parent-child relationship allows the child KPI value to roll up the parent KPI. For the Police Department KPIs, the value of Crime Response Time Precinct One and Crime Response Time Precinct Two determine the value of Crime Response Time, and the value of Crime Response Time determines the value of Police Department. If, for example, the value of Crime Response Time Precinct One signifies "take action" then the value of Crime Response Time will also signify "take action".

Figure 1. Police Department KPIs
Police Department KPIs

The top level parent KPI within a given KPI tree is referred to as a level 1 KPI, its children are referred to as level 2 KPIs, and so on. In Figure 1, Police Department is a level 1 KPI, Crime Response Time is a level 2 KPI, and Crime Response Time Precinct One and Crime Response Time Precinct Two are level 3 KPIs. The Public Safety KPIs and their associated levels are shown in the table below.

Table 1. Public Safety KPIs
Level 1Level 2Level 3
Fire DepartmentFirefighter injuriesFirefighter injuries Fire Station One
Firefighter injuries Fire Station Two
Police DepartmentCrime Response TimeCrime Response Time Precinct One
Crime Response Time Precinct Two
Public Safety managementPublic Safety budgetEMS Department budget
Fire Department budget
Police Department budget

Creating an IBM Intelligent Operations Center KPI

In the rest of this article focuses on IBM Business Monitor components and the general process for creating a new Intelligent Operations Center KPI. To help illustrate the concepts, we will use the Crime Response Time Precinct One KPI as our example.

To create a new KPI you must first determine its requirements. What information should the KPI convey? What CAP message or messages contain the information required to calculate the KPI? What is the period of time covered by the KPI? Defining the KPI requirements is often the hardest task when creating a new KPI. After you've defined the KPI requirements, you use the IBM Business Monitor development toolkit to create the KPI. The requirements for the Crime Response Time Precinct One KPI are:

  • Calculate the average response time to crimes where the reported severity is severe or extreme.
  • Calculate the KPI over a seven-day period.
  • Calculate the KPI for Precinct One.

CAP messages with an event type of Crime_Response_Time contain the information required to calculate the average crime response time for a given police precinct. Figure 2 shows a sample Crime_Response_Time CAP message. The <severity> element indicates the severity of the reported crime; the <onset> element contains the date the crime was reported; the first <parameter> segment contains the report number; the second <parameter> segment contains the reporting precinct; the third <parameter> segment contains the date the precinct responded to the reported crime.

Figure 2. Crime_Response_Time CAP message
Crime_Response_Time CAP message

Monitor models

The first step in creating a new KPI is to create or use an existing monitor model to house the KPI definition. The monitor model is a top-level container that contains the monitor details model, the KPI model, the dimensional model, the visual model, and the event model. The Intelligent Operations Center sample monitor models use the monitor details, KPI, and event models.

The monitor details model contains the monitoring context definitions. A monitoring context defines the inbound events (events sent to the Business Monitor server), metrics (data) extracted or derived from inbound events, outbound events (events sent from the Business Monitor server), conditions for filtering events, stopwatches, counters, and triggers. Run time instances (monitoring context instances) use inbound events to monitor an entity, for example, a particular process execution, the state of a particular order, or the stock level of an item in a warehouse. A monitoring context is created for each real-world entity that is monitored. For the Intelligent Operations Center, the monitored entities are CAP messages. The information collected at run time from CAP message is the data required to calculate a KPI.

The KPI model contains the KPI context definitions. A KPI context defines the key performance indicators (KPIs) and their associated triggers and events. KPI contexts can process inbound events, evaluate recurring wait-time triggers (for example, checking whether a KPI has gone out of range), and send outbound events (for example, a notification indicating the KPI is out of range).

The event model is the component that refers to all event inbound and outbound definitions used in the monitor model. It contains references to the schemas used to describe the structure of individual event parts.

Monitor models are contained in Business Monitoring projects. Wizards for creating Business Monitoring projects, monitor models, and the other required Business Monitor components are installed into Rational Application Developer or WebSphere Integration Developer when you install the IBM Business Monitor toolkit. The pattern used in the Intelligent Operations Center sample KPIs is to use a single Business Monitoring project with separate monitor models for related sets of KPIs.

When you create a monitor model, the model XML is displayed in the monitor model editor allowing you to use the editor views, forms, and functions to create the model as opposed to editing the XML directly.


Monitoring context

After you've created the monitor model you're ready to define the monitoring context. The monitoring context is where you define the CAP messages (inbound events) sent to the Business Monitor server, the metrics extracted or derived from the messages, conditions for filtering the messages, and correlation expressions that determine whether a new monitoring context instance is created or an existing instance is used.

The pattern used in the Intelligent Operations Center sample monitor models is to create separate monitoring contexts for each CAP message sent to Business Monitor. The monitoring context naming convention used in the sample monitor models uses the CAP event type without the underscores as the context name. Using our example CAP message, we would create a monitoring context called Crime Response Time as illustrated in Figure 3.

Figure 3. Monitor model monitoring contexts
icoc sample public safety monitor model monitoring contexts

Inbound events

Inbound events deliver the information required to calculate a KPI to the Business Monitor server. Inbound events and other monitoring context elements can be created using the monitoring context pop-up menu, as shown in Figure 4.

Figure 4. Monitoring context pop-up menu
Monitoring context popup menu

The inbound event naming convention used in the Intelligent Operations Center sample monitor models combines the CAP event type without the underscores with the string "Message". Using our example CAP message, we would create an inbound event called Crime Response Time Message. See Figure 5.

Figure 5. Crime Response Time Message inbound event
Crime Response Time Message inbound event

Connecting the inbound event to the CAP XML schema

After defining the inbound event, you will need to specify the CAP XML schema definition to Business Monitor so that it knows the CAP message structure. The schema must exist in the Business Monitoring project containing the monitor model. Schema definitions are specified using the monitor model editor Events Type Details view. The CAP XML schema (CAP-v1.2-os.xsd) is included in the sample monitor models. To specify the schema you'll use Events Type Details add function and reference the CAP-v1.2-os.xsd file. See Figure 6.

Figure 6. Event Type Details view
Event Type Details

Inbound event filtering

The inbound event filter condition is an expression that evaluates to true or false. If the expression evaluates to true, the inbound event is sent to the monitoring context. If the expression evaluates to false, the inbound event is discarded. Based on our example KPI definition, we only want to process CAP messages with an event type of Crime_Response_Time and that have a severity value of extreme or severe. To filter out messages that don't meet these criteria you would specify a filter condition similar to one shown in Figure 7. The filter condition converts the CAP <event> and <severity> element values to lower case and compares them to the strings "crime_response_time", "severe", and "extreme". CAP messages failing this test are not sent to the monitoring context, that is, they are discarded.

Figure 7. Inbound event filter condition
Inbound event filter condition

Note: When creating expressions in any of the monitor model editor form views, you can press Ctrl+Space for content assist. Press Ctrl+Space to bring up the content assist window. The content assist window consists of model elements, operators, and functions; both built-in functions and user-defined XML Path Language (XPath) functions. Selecting an item from the tree and pressing Enter places the XPath 2.0 representation of that item (or reference to that item) in the expression at the cursor position. The content assist window is sensitive to the expression that you are editing, and, when possible, shows only valid options.

Figure 8. Content assist
Content assist

Monitoring context instance

The concept of a monitoring context instance is sometimes difficult to grasp. A monitoring context instance represents a single instance of the metrics collected in a monitoring context. For the Intelligent Operations Center, a monitoring context instance is somewhat analogous to a CAP message. When a CAP message comes in, a monitoring context instance is created or refreshed, and metrics in the context instance are populated with values from the CAP message. Note the term "refreshed". It is possible to define a monitoring context where only one instance of the context is created. In this situation, metrics in the context instance are refreshed with values from the CAP message whenever a new message is sent to the Business Monitor server.

The KPI definition determines if you need to create a new monitoring context instance for each CAP message. In our example we want to calculate the average response time to reported crimes. Since a Crime_Response_Time CAP message contains information about the response to a single crime, we will need to create a monitoring context instance for each Crime_Response_Time CAP message processed by the Business Monitor server. The KPI will be calculated (averaged) using the metrics from each monitoring context instance.

Monitoring context key

Each monitoring context requires at least one key. The monitoring context key is used to uniquely identify a monitoring context instance. The key is assigned to the monitoring context instance when the instance is created. In our KPI example, we need to create a monitoring context instance for every Crime_Response_Time CAP message received. As a result, we need to choose a key that will be unique for each message. The CAP identifier element uniquely identifies a CAP message so we can use this value as our key. The key expression for the Crime Response Time monitoring context is shown in Figure 9.

Figure 9. Monitoring context key
Monitoring context Key

Inbound event correlation expression

The correlation expression is used to determine if a monitoring context instance (or instances) exists. Often it is simply attempting to match an inbound event attribute with the monitoring context key. If the expression evaluates to true, then one or more instances exists. If the expression evaluates to false, then no instance was found. The correlation expression has three options that allow you to specify what to do with the inbound event if no monitoring context instance is found, if a single instance is found, or if multiple instances are found. The correlation expression defined for the Crime Response Time Message inbound event is shown in the Figure 10. The expression checks to see if the monitoring context key is equal to the CAP message identifier. This expression should always evaluate to false since the message identifier for each CAP message is unique. The correlation expression options are set to create a new instance if no instances are found and treat all other conditions as an error.

Figure 10. Correlation expression
Correlation expression

Monitoring context metrics

Metrics hold information for a monitoring context instance. KPI aggregation functions use metrics to calculate KPI values. For the Intelligent Operations Center, metrics hold information passed to the Business Monitor server in CAP messages. When the Business Monitor server receives a CAP message, it creates or reuses an existing monitoring context instance and populates the metrics defined for the context with values extracted or derived from the CAP message.

Metrics have an associated data type and a value expression. The data type is specified in the Metric Details view. This view is available when you create or select a metric. The value expression tells Business Monitor how to get or derive the metric value. Metric value expressions are defined using the Metric Value Expressions view. For most metric definitions, the metric value expression specifies the CAP element containing the metric value.

The metric value expression for the response date metric is shown in Figure 11. The expression selects the <value> element from the third <parameter> segment of the first <info> segment and converts the value to a DateTime value using the xs:dateTime() XPath function.

Note: Limitations in Business Monitor's XPath support combined with the allowance of multiple info and parameter blocks requires that you specify the specific <info> and <parameter> segments containing the element values used to populate the metric. This is important when defining the monitoring context and requires that parameters in the incoming CAP message be in the correct order.

Figure 11. Metric value expression
Metric value expression

The complete set of metrics defined for the Crime Response Time monitoring context and the elements they map to in the Crime_Response_Time CAP message are shown in Figure 12.

Figure 12. Metrics
Metrics

As you can see, all metrics map to elements passed in the Crime_Response_Time CAP message except for response time. This metric is calculated by subtracting the date metric from the response date metric. The completed metric value expression would be: response_date - date. You can use the content assist window to select the metric IDs and operator or enter the expression directly into the form view.

Note: Business Monitor element IDs are used in expressions. IDs cannot contain spaces or special characters. The convention used in the Intelligent Operations Center sample monitor models is to use spaces in the element names and underscores in place of spaces in element IDs.

Figure 13. Response time metric value expression
responee time metric value expression

Context completion trigger

The last element defined for the monitoring context is a trigger to terminate the monitoring context instance. Triggers are control points used to initiate (trigger) additional processing within Business Monitor. Triggers have a source and a condition that determine when the trigger fires. The source (for example, inbound event) initiates the processing and the condition determines if the trigger should be fired.

The decision to terminate a monitoring context instance is based on the KPI requirements. If a new monitoring context instance is created for each incoming CAP message then the monitoring context instance can be terminated. If the KPI requires a single monitoring context instance where the metric values are "refreshed" whenever a CAP message is received, the monitoring context instance should not be terminated.

Our example KPI requires a monitoring context instance for each Crime_Response_Time CAP message. As a result, we can terminate the monitoring context instance after the CAP message has been received and the instance created. The trigger to terminate the monitoring context instance is shown in Figure 14. The source for the Completion trigger is the Crime Response Time Message inbound event. The trigger condition checks to see if the monitoring context key is equal to the identifier metric and if it is, the trigger is fired and the monitoring context instance is terminated. This trigger definition essentially terminates the monitoring context instance as soon as the Crime_Response_Time CAP message is processed.

Figure 14. Completion trigger
Completion trigger

Completed monitoring context

The complete Crime Response Time monitoring context is shown in Figure 15.

Figure 15. Completed Monitor Details Model
Completed Monitor Details Model

KPI context

Now that we completed the monitoring context definition, we are ready to define the KPI. KPIs are defined within a KPI context so the first thing we need to do is create the KPI context. KPI contexts are defined in the KPI model. To switch to the KPI model, you click on the KPI Model tab in the monitor model editor.

The pattern used in the Intelligent Operations Center sample monitor models is to create a separate KPI context for each level 1 KPI. In our example, Police Department is the level 1 KPI. The KPI context naming convention used for in the sample monitor models is to append the string "KPIs" to the KPI name. For our example, we would create a KPI context called Police Department KPIs as shown in Figure 16.

Figure 16. KPI Contexts
KPI Contexts

Crime Response Time Precinct One

Now it is time to define the Crime Response Time Precinct One KPI. KPIs and other KPI context elements can be created using the KPI context pop-up menu. (See Figure 17.) Because the default Intelligent Operations Center configuration displays the KPI name in the KPI portlets, you will want to use a meaningful name when creating a KPI.

Figure 17. KPI context pop-up menu
KPI context popup menu

KPIs have an associated data type. The data types are Decimal and Duration. When you create a KPI the data type is set to Decimal. The Duration type is used to specify a time interval. Since we want our example KPI to calculate the average crime response time, we will need to change the data type to Duration. The data type is specified in the KPI Events Detail view in the monitor model editor when you select the KPI. (See Figure 18.)

Figure 18. KPI Details view
KPI Details view

KPI ranges

KPIs can have a set of ranges, each of which defines a span of possible values. Ranges can be specified either as a percentage of the target value or as an actual value. For each KPI range, you specify a start and end value, a color, a name, and an ID. Ranges are specified in the KPI Target and Ranges view, which is displayed in the monitor model editor when you create or select a KPI. Most of the Intelligent Operations Center sample KPIs have three defined ranges. The range values vary for each KPI but the names and colors are the same. The range name and colors used are: acceptable (green), caution (yellow) and take action (red). Consistent range names and colors are used because the range name and color RGB values are used by the Intelligent Operations Center KPI portlets to represent a KPI's status. The Crime Response Time Precinct One KPI range definitions and the KPI Drill Down portlet are shown in Figure 19 and Figure 20. In this example, the Crime Response Time Precinct One KPI value falls within the acceptable range. The KPI status and color are set to the range name (acceptable) and color (e4f2d1) in the KPI Drill Down portlet.

Figure 19. KPI range definitions
KPI range definitions
Figure 20. KPI Drill Down portlet
KPI Drill Down portlet

KPI definition

KPIs get their values in one of the following two ways: the value comes from a metric (defined in the monitoring context) using an aggregation function, or the value comes from a calculation based on other KPIs or user-defined XPath functions. KPIs that get their value from metrics and aggregation functions are referred to as aggregation KPIs. KPIs that get their values from other KPIs or user-defined XPath functions are referred to as expression KPIs. The pattern used by the Intelligent Operations Center sample KPIs is to define the lowest level KPIs (that is, KPIs with no children) as aggregation KPIs and the higher level KPIs (that is, KPIs with children) as expression KPIs. Aggregation KPI values are expressed as quantifiable measurements such as 5 minutes, 7 seconds, or 100.5.

The KPI calculation type (aggregation or expression) is specified in the KPI Definition view. This view is displayed in the monitor model editor when you create or select a KPI. The Crime Response Time Precinct One KPI is defined as an aggregation KPI because it has no children. When defining an aggregation KPI, the KPI Definition view displays two addition sub-views. The KPI Details sub-view is where you select the monitoring context, metric, and aggregation function used in the KPI calculation. The Time Filter sub-view is where you specified the time period for the KPI calculation. The KPI definition for the Crime Response Time Precinct One is shown in Figure 21. The Crime Response Time monitoring context and the response time metric provide the input values to average aggregation function. The KPI time period is defined as a rolling 7-day period. The aggregation function and time period specified in the KPI definition are determined by the KPI requirements described earlier in this article.

Figure 21. KPI definition
KPI definition

The Crime Response Time Precinct One KPI definition is now complete.

KPI data filter

The last thing we need to define is the KPI data filter. Data filters provide a means for limiting the monitoring context instances used in a KPI calculation. We need to define a data filter because we want our KPI to calculate the average crime response time for Precinct One but not for any other precincts. When defining a data filter you select a metric from the monitoring context used by the KPI aggregation function, an operator, and a set of values to be used in a comparison operation with the specified metric. The KPI data filter defined for the Crime Response Time Precinct One KPI is shown in Figure 22. This data filter compares the precinct metric to the string "precinct one". The Case-sensitive option is unchecked to ensure the filter is not case sensitive. Only monitoring context instances that pass this data filter will be used in the KPI Calculation.

Figure 22. KPI data filter
KPI data filter

Conclusion

After examining how our a sample KPI was modeled you should you have a better understanding of how to develop a low-level KPI.

In the next article we will describe how the remaining KPIs (that is, the parent KPIs) in the KPI tree are modeled and configured in the IBM Intelligent Operations Center. We will also look at how asynchronous KPI updates are triggered using outbound events defined in the KPI context.

Resources

Learn

Get products and technologies

  • IBM product evaluation versions by industry: Find the products used in each Industry Framework. Industry Frameworks are specific software platforms designed to harness the power of IBM products to meet industry standards.

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Industries
ArticleID=710596
ArticleTitle=IBM Intelligent Operations Center key performance indicators (KPIs), Part 1: Defining a low-level KPI
publish-date=07122011