Deliver actionable business insight with low latency analytics using WebSphere eXtreme Scale and Cognos Real-time Monitoring

The use of data analytics promises to deliver a wealth of opportunities that will enable game-changing advances in the level of achievable business insight across all domains of computing applications. As such, organizations are being urged to implement integrated information platforms that support multiple modes of data analysis. This article shows how data written to an IBM® WebSphere® eXtreme Scale-based distributed cache can be consumed by IBM Cognos® Real-time Monitoring, thereby enabling low latency analytical processing. This content is part of the IBM WebSphere Developer Technical Journal.

Share:

Dr. Alan Hopkins (hopkinsa@uk.ibm.com), Senior IT Specialist, IBM

Alan is a Consulting IT Specialist with IBM Software Group Services for WebSphere with more than 20 years experience in middleware and internet-related technologies. For much of his career he has focused on Transactional Middleware systems, and their usage in the e-Business space. Most recently Alan has been dedicated to the strategically important areas of Business Process Management/Business Activity Monitoring and Extreme Transaction Processing. At the present time Alan is a member of the Worldwide Technical Practice, who form part of IBM Software Services for WebSphere, based at IBM's Hursley Park Laboratory in England.



08 May 2013

Also available in Chinese

Introduction

The potential benefits that could be realised through the analysis of exponentially increasing volumes of data generated by our increasingly instrumented and interconnected world is causing a great deal of excitement. Much of this excitement has focused on the promise of Big Data — the analysis of extremely large data volumes, typically using batch-oriented approaches to process data residing on disk, distributed across multiple physical machines. Although the potential of these increasingly popular batch-oriented analytical techniques is undoubtedly exciting, we should not overlook the power of lower-latency analytical approaches to complement their batch counterparts.

Indeed, Figure 1, which presents the idea of an Information Half Life (see Resources) suggests that the value that can be derived from data falls rapidly over time from the point of creation or change. Organizations wishing to leverage the benefits of data analytics are being urged to build a coherent and integrated information platform that supports the analysis of data across the entire latency spectrum.

Figure 1. Information Half-Life
Information Half-Life

In recent years, many organizations have introduced a discrete caching layer into their technical infrastructure as a means to facilitate the scale-out of applications to meet ever increasing concurrency and response time demands. IBM WebSphere eXtreme Scale provides a coherent distributed caching facility that is able to scale linearly to support access to large volumes of cached data. Data residing within this cache can be leveraged by analytical tools to enable near-real-time insight into business opportunities and threats. As such, this capability can provide a fundamental pillar in an enterprise integrated information platform.

This article presents the implementation details of a near-real-time analytical scenario that leverages the distributed caching capabilities of WebSphere eXtreme Scale together with the analytical functionality provided by Cognos Real-time Monitoring. The scenario aims to demonstrate how the complementary capabilities of these two products might be leveraged to fulfill a core requirement of an enterprise integrated information platform.

The scenario described here was implemented using the following software:

  • IBM Cognos Real-time Monitoring Version 10.2
  • IBM WebSphere Application Server Network Deployment V7.0
  • IBM WebSphere eXtreme Scale Version 8.6
  • IBM DB2 Database Version 9.7.

Low latency analytics scenario

The simple use case presented here is based around a fictitious financial services scenario that requires a near-real-time analytical reporting capability of metrics related to stock trades. A high level overview of the major components in the scenario is presented in Figure 2.

Figure 2. Scenario overview
Scenario overview

In a real life implementation, the processing of stock trades is managed by one or more enterprise application suites. In order to reduce critical data retrieval latency, many organisations introduce a distributed caching layer into their middleware landscape. Events encapsulating business critical data are propagated to the cache, from where it can be efficiently retrieved by consumers.

The focus of this article is to show how data propagated to the caching layer of an enterprise middleware landscape can be leveraged to provide the data underpinning the presentation of critical data via a business dashboard. As such, the propagation of data to the cache is done via the use of a simple program.

Aggregation and rendering of data in the form of visual widgets to produce a business dashboard view is handled by Cognos Real-time Monitoring. Data can be propagated to Cognos Real-time Monitoring in a variety of ways – here we use a web-services based approach. From a Cognos Real-time Monitoring perspective a web service mechanism is provided to enable data to be pulled from a service endpoint at a configurable time interval. Alternatively, data can be pushed from a data source via a web service interface exposed by Cognos Real-time Monitoring. We exploit this latter capability in the implementation of our example scenario.

Data written to WebSphere eXtreme Scale is pushed to Cognos Real-time Monitoring via the use of the commonly used write-through architectural pattern. The implementation of this pattern is supported through the use an architected code extension point provided by WebSphere eXtreme Scale. Here, specific methods encapsulated by plugin code modules are invoked at key points during cache execution.


Scenario implementation details

Let's look at the scenario implementation. Providing detailed, click-by-click instructions that might be used to build this scenario independently is beyond the scope of this article. However, we'll take a detailed look at the various artifacts that collectively comprise the scenario. A zip file containing scenario artifacts is available with this article for download.

Cognos Real-time Monitoring configuration

Let's begin with the configuration of Cognos Real-time Monitoring, a component of the IBM Cognos Business Intelligence suite that is specifically designed to enable low-latency analytical reporting of business critical data via configurable dashboards. Cognos Real-time Monitoring provides a range of visual widgets from which dashboards can be assembled. In this scenario, a number of these widgets are used to illustrate the breadth of available capabilities, as summarised in Table 1.

Table 1. Cognos Real-time Monitoring widgets
Widget type Purpose
Bar Chart Total hourly stock volumes
Bar Chart NASDAQ hourly stock volumes
Bar Chart DJIA hourly stock volumes
Bar Chart FTSE100 hourly stock volumes
Drillable Bar Chart Trade values – drillable by market, region and sector
Pie Chart % Trade types – Buy/Sell/Short
Alert Triggered following a trade with value in excess of $1,000,000
Speedometer Average trade execution time relative to SLA ranges

Inbound DataStream definition

To enable Cognos Real-time Monitoring to accept data pushed from an external source via SOAP/HTTP you must define an inbound DataStream of type Web Service. The fields that you need to be populated in the inbound data can be specified at DataStream definition time; the fields that are present in our example scenario are shown in Figure 3.

Figure 3. Datastream fields
Datastream fields

In order to propagate stock trade data to Cognos Real-time Monitoring via the web service interface exposed by StockTradeDataStream you need to get hold of the WSDL interface specification. This can be retrieved by accessing the following URL from a web browser:

http://<hostname:port>/realtime/wsdl/eventstream.wsdl

where <hostname:port> refers to the Cognos Real-time Monitoring HTTP host name and port number.

Save the exported WSDL interface definition to your file system. This can be used to generate a web service client that propagates stock trade events to Cognos Real-time Monitoring. The generation of a web service client from a WSDL service definition is outside the scope of this article and hence will not be covered in detail here. Note that the generated client stub will be used to enable the propagation of cache update events from WebSphere eXtreme Scale to Cognos Real-time Monitoring, and will be described later in this article.

Trade volumes view definitions

Inbound data consumed by a Datastream is made available for displaying in a Dashboard object via a View. The View construct enables you to compute measures from aggregate expressions across multiple inbound events. For example, you want to present overall trade volumes per hour as a bar chart in your business dashboard, and hence you need a View based upon your inbound Datastream that computes the aggregate measures that underpin this Dashboard object. The Hourly Trade Volumes view is shown in Figure 4.

Figure 4. Create view
Create view

The expressions used to compute the Time_Hour and Hourly Volume metrics, respectively, are summarised in Table 2.

Table 2. Metrics expressions
Metric Expression
Time_Hour SUBSTRING (StockTrades.DataStreams.StockTradeDataStream.tradeTime, 12, 2)
Hourly Volume SUM(StockTrades.DataStreams.StockTradeDataStream.volume)

See also that the Group By option is checked for Time_Hour; this specifies that volumes are summarised per hour.

Three additional views are also defined to enable the presentation of Hourly trade volumes specific to the Nasdaq, Dow Jones, and FTSE100 exchanges. These views were defined in a similar manner to the Hourly Trade Volumes View outlined above, with an additional Where filter clause resembling Figure 5.

Figure 5. Additional views
Additional views

Trade volumes business dashboards

The Cognos Real-time Monitoring Workbench enables you to define a Business Dashboard Object based upon the Trade Volumes Views. A number of visual object types are available so that data can be presented in different ways. In Figure 6, for example, Hourly Trade Volumes are represented as a column based chart.

Figure 6. Trade Volumes
Trade Volumes

Dimensional analytics

A dimension can be viewed as a multi-level filter argument that can be used to facilitate the analysis of data at varying levels of scope. Computed data measures are combined with one or more predefined dimensions to form a multi-dimensional data cube. Business Dashboard views are then defined upon these cubes as a means of visualisation. In this way, end-users are able to view data via a suitable visual metaphor at the most general dimension level, and to progressively drill down to view data at an ever increasing level of detail.

The Business Dashboard here will feature a drillable bar chart that will enable the user to dynamically drill down total stock trade volumes to reveal data values by region, market, sector to individual stock level. A number of artifacts must be defined within Cognos Real-time Monitoring in order to define a cube-based dimensional analysis capability. A high level description of the required artifacts is provided below.

Metadata defining dimensions and their hierarchical levels can be provided in one or more lookup tables. Within Cognos Real-time Monitoring, access to these tables is provided by a JDBC agent. Figure 7 presents details of our JDBC agent.

Figure 7. JDBC agent
JDBC agent

The hierarchical levels are defined in a lookup table that is accessed via the JDBC agent. By way of example, Figure 8 shows a snapshot of how the Stocks table is populated (only the first three rows are shown here).

Figure 8. Stock table
Stock table

The corresponding definition of the Cognos Real-time Monitoring Lookup Table is presented in Figure 9.

Figure 9. Stocks Lookup Table
Stocks Lookup Table

Once the underlying Lookup Table has been implemented, it is possible to define a Dimension. The ordering of levels within the Stocks Dimension that will be used here is shown in Figure 10.

Figure 10. Ordering levels
Ordering levels

Finally, you can define a Cube that encapsulates the aggregated metrics that you ultimately wish to present via your Business Dashboard, together with the Dimensions that will be available for the user to specify as the basis for drilling down through the data.

Figure 11. Cognos Real-time Monitoring Cube
Cognos Real-time Monitoring Cube

Be aware that although this Cube definition comprises two aggregate Measures, together with the single multi-level Stocks dimension that will be available to assist dynamic data analysis, it is possible to define additional dimensions within a cube.


Drillable business dashboard object

In this example, the dimensional data will be rendered as a Bar Chart dashboard object.

Figure 12 illustrates how stock trade volume data can be displayed at differing levels of your defined dimension. At the highest level, a single bar represents the total volume of all stocks. Drilling down, you can view volume for each defined region, Europe or USA, and you can continue to drill down through market and sector to individual stock.

Figure 12. Dimensional drilldown
Dimensional drilldown

Trade Types pie chart

Once you have the inbound Datastream construct defined, you can easily add further dashboard objects to illustrate the use of other presentation styles. Next, we’ll examine the addition of a pie chart dashboard object to visualise the percentage of Buy vs Sell vs Short stock trade types.

A pie chart dashboard object requires two types of measure to be specified at configuration time. The first is the Value measure, which provides the numbers to represent; the other is the category that will be used to determine the sectors of the pie chart. In this dashboard object, the Count and Trade Type measures are used as the Value and Category parameters respectively. Figure 13 presents the definition of these measures, encapsulated within a dedicated View.

Figure 13. Trade types view
Trade types view

The corresponding pie chart showing the relative frequency of stock trade types as percentages is shown in Figure 14.

Figure 14. Trade types pie chart
Trade types pie chart

High value trade alert

Cognos Real-time Monitoring enables the definition of Alerts that will be triggered when a user-defined condition occurs. In this scenario, an Alert is configured to be triggered when the value of any single trade exceeds a threshold value of $1,000,000. First, you define a View that will be subsequently used to determine whether the Alert is triggered. This View encapsulates data fields that will be used to determine whether the Alert is triggered, and to populate the message embedded within a triggered Alert.

Figure 15. Alert
Alert

Unlike the earlier constructs that we have considered, Alerts are configured in the Scenario Modeler tab of the Cognos Real-time Monitoring Workbench. The definition and subscription details for this Alert are configured as shown in Figure 16. (Here, Alert visibility is configured only for user rtmadmin, and the Body of the Alert message features values derived from the TradeValue view.)

Figure 16. Alert detail
Alert detail

The conditions under which a given Alert is triggered are configured by associating the Alert with a Business Rule. Once again, the value is derived from TradeValue view defined specifically for this purpose.

Figure 17. Business rule for alert
Business rule for alert

Trade Execution Time SLA Speedometer

Cognos Real-time Monitoring provides a selection of Indicator Dashboard Object types, that enable a numeric measure to be represented on a scale. These Indicators can be customised at definition time, enabling a current value to be represented against predefined target and range values. In this example, a Speedometer indicator is used to visualise the average trade execution time value against predefined ranges from a fictitious Service Level Agreement.

An additional View is defined here to hold Trade execution time related measures, based upon StockTradeDataStream, discussed earlier. The average trade execution time visualised in the Speedometer is computed from this expression:

AVG(StockTrades.DataStreams.StockTradeDataStream.duration)

Three ranges are specified to show Good, Average and Poor execution time SLA ranges at Dashboard Object definition time (Figure 18).

Figure 18. Trade execution time range
Trade execution time range

The resulting Speedometer indicator is shown in Figure 19.

Figure 19. Trade execution time speedometer
Trade execution time speedometer

WebSphere eXtreme Scale configuration

Now, let's take a look at how the WebSphere eXtreme Scale cache is configured. There are four distinct areas of interest here:

  • Structure of Stock Trade cache entry
  • WebSphere eXtreme Scale client program
  • Definition of WebSphere eXtreme Scale cache
  • Propagation of cache updates to Cognos Real-time Monitoring.

Let's start by examining the Java value object that is used to encapsulate interesting details of the Stock Trade event. The attributes that are defined to hold the data of interest are shown in Listing 1. (The value object also contains getter and setter methods, but these have been omitted for brevity.)

Listing 1. WebSphere eXtreme Scale cached object definition
public class StockTrade implements java.io.Serializable
{
	String			ticker		= null;		 
   	BigDecimal		price 		= null; 
    	BigInteger		volume		= null;
    	BigInteger		duration 	= null;
   	String 			market		= null;
    	String 			type 		= null;
    	String 			sector		= null;
    	XMLGregorianCalendar 	tradeTime 	= null;
	
	.
	.
	.
}

A simple WebSphere eXtreme Scale command line Java program can be used to simulate the emission of Stock trade events for this example. In an actual scenario, these events would be emitted by enterprise operational systems, however the final insertion into the WebSphere eXtreme Scale cache could well leverage similar client API calls to those illustrated in Listings 2 and 3.

Listing 2. Simulated WebSphere eXtreme Scale data source
public class WXSStockTradeClient {
	
   String cep 		= "localhost:3809,localhost:3810";
   String gridName 	= "AnalyticsGrid";
   String mapName 	= "StockTrades";

   public static void main(String[] args) 
   {
	WXSStockTradeClient client = new WXSStockTradeClient();
		
	client.execute("IBM",100.0f,1000,12,"DJIA","Buy","Technology");

	.
	.
	.
   }

The execute() method instantiates the StockTrade value object and inserts into the cache using the ObjectMap upsert() API call.

Listing 3. Simulated WebSphere eXtreme Scale data source (continued)
void execute(String ticker, float price, int volume, int duration, 
                          String market, String tradeType, String sector)
{
	StockTrade trade = new StockTrade();
		
	trade.setTicker(ticker);
	.
	.
	.
		
	try
	{
		og = getObjectGrid(cep,gridName);
		Session session = og.getSession();
		ObjectMap cache = session.getMap(mapName);
			
		session.begin();		
		cache.upsert(ticker, trade);
		session.commit();
		.
		.
		.
}

(In the above, the initial client bootstrap to the WebSphere eXtreme Scale cache is encapsulated inside the getObjectGrid() method, as shown in Listing 4.)

Listing 4. Get Handle to ObjectGrid
synchronized static public ObjectGrid getOG(String cep, String gridName)
{		
   if (og == null)
  {
	try
	{		
	   ClientClusterContext ccc =
                      ObjectGridManagerFactory.getObjectGridManager().
connect(cep,null,null);
					
	   ObjectGridManager ogm = ObjectGridManagerFactory.getObjectGridManager();
			
	   og = ogm.getObjectGrid(ccc,gridName);			
	}
	catch (Exception e)
	{
		System.out.println("Exception caught in 
WXSStockTradeClient.getOG() - " + e);
		System.out.println("Message = " + e.getMessage());
		System.out.println("Cause = " + e.getCause());
		e.printStackTrace();
	}			
  }		
  return og;
}

Propagation of cache updates to Cognos Real-time Monitoring

A custom Loader plugin is used here to enable you to propagate cache changes to Cognos Real-time Monitoring. When a plugin of this type is configured for a WebSphere eXtreme Scale Grid, the BatchUpdate() method is invoked to enable updates to be propagated to external systems. Details of the committed API calls together with key and value objects are provided via the parameter list. The frequency of BatchUpdate() invocation is configurable. In this example, updates are propagated to Cognos Real-time Monitoring within the transaction that commits changes to the cache, thereby ensuring minimal latency. However, in some situations it may be preferable to batch updates together for propagating externally. This "write-behind" caching pattern is often used in order to reduce the workload on external endpoint systems, and should be considered for use when some latency in propagating cache updates to the external system is acceptable.

The essential code elements of the BatchUpdate() method are presented in Listing 5. For insert(), put() and upsert() calls, our custom code invokes the propagateToCRTM() method.

Listing 5. WebSphere eXtreme Scale custom loader plugin
public void batchUpdate(TxID tx, LogSequence sequence)
			throws LoaderException, OptimisticCollisionException 
{
    try 
    {
        Iterator iter = sequence.getPendingChanges();
	  while (iter.hasNext()) 
	  {
	      LogElement logElement = (LogElement) 
iter.next();
	      String key = (String)logElement.getKey();
	      StockTrade value = (StockTrade)logElement.
getCurrentValue();
	                
	      switch (logElement.getType().getCode()) 
	      {	                    
	           	case LogElement.CODE_UPSERT:
	          		propagateToCRTM(key, value);
	                  break;	        
	       }
	    }           
	} 
      catch (Exception e) 
      {
	            LoaderException ex = new LoaderException(e);
	            throw ex;
	}
}

The propagateToCRTM() method encapsulates a call to the web service endpoint that represents your inbound to Cognos Real-time Monitoring DataStream, and which is described by the WSDL exported from Cognos Real-time Monitoring earlier.

Listing 6. WebSphere eXtreme Scale custom loader plugin (continued)
void propagateToCRTM(String key, StockTrade value)
{		
   PublishEventSoapProxy proxy = new PublishEventSoapProxy();		
   String endpoint = "http://localhost:9090/realtime/webservice/eventstream";
   
   Descriptor descriptor = proxy._getDescriptor();
   descriptor.setEndpoint(endpoint);
		
   StockTradesDataStreamsStockTradeDataStreamInput input = new
                                   StockTradesDataStreamsStockTradeDataStreamInput();
		
   EventInfoType eventInfo = new EventInfoType();
   eventInfo.setEventname("StockTrades.DataStreams.
StockTradeDataStream");
   eventInfo.setUsername("rtmadmin");
   eventInfo.setPassword("manager");		
   input.setCqesi(eventInfo);
		
      List sqData = input.getStockTradesDataStreamsStockTradeDataStreamData();
		
   StockTradesDataStreamsStockTradeDataStreamDataType trade = new
                                StockTradesDataStreamsStockTradeDataStreamDataType();
   trade.setTicker(value.getTicker());
   .
   .
   .
   sqData.add(trade);
		
   try
   {
	proxy.
publishDataStockTradesDataStreamsStockTradeDataStream(input);
   }
   catch (Exception exc)
   {
	System.out.println("Exception caught ...." + exc.getCause());
   }
}

Business Dashboard

Finally, it is possible to compose a Business Dashboard from many of the dashboard objects presented earlier (Figure 20).

Figure 20. Full dashboard
Full dashboard

When a series of Stock Trade objects are injected into a scenario in which a number of trades have been defined to exceed the high value trade Alert threshold, a set of corresponding alerts will be triggered. After logging in and accessing the Dashboard, an Alert Manager indicator button appears above the main Dashboard area (Figure 21).

Figure 21. Alert indicator
Alert indicator

A flashing red exclamation mark informs the Business Dashboard user that alerts have been triggered. Clicking on the Alert Manager button reveals an informational message and a further link to the Alert Manager, where further details of the generated Alerts can be viewed (Figure 22).

Figure 22. Alerts
Alerts

Conclusion

This article showed how data written to a WebSphere eXtreme Scale cache can be consumed by Cognos Real-time Monitoring. In this way, in-flight data can be leveraged to deliver low-latency analytics-based insight via a Business Dashboard-based presentation approach. The use of near real-time analytics can supplement other higher-latency analytical techniques as part of a comprehensive enterprise information integration platform.


Acknowledgements

Many thanks to Greg Reid for his review and provision of useful feedback for this article.


Download

DescriptionNameSize
Code sampleSupportingArtefacts.zip294 KB

Resources

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into WebSphere on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=WebSphere, Big data and analytics
ArticleID=915437
ArticleTitle=Deliver actionable business insight with low latency analytics using WebSphere eXtreme Scale and Cognos Real-time Monitoring
publish-date=05082013