IBM Accelerator for Machine Data Analytics, Part 1: Speeding up machine data analysis

Machine logs from diverse sources are generated in an enterprise in voluminous quantities. IBM® Accelerator for Machine Data Analytics simplifies the task of implementation required so analysis of semi-structured, unstructured or structured textual data is accelerated.

Share:

Sonali Surange (ssurange@us.ibm.com), Software Architect, IBM

Author photoSonali Surange is an IBM Software Architect working on IBM's big data products and technologies. She has filed numerous patents, published over 15 technical papers with IBM developerWorks, and presented in numerous technical conferences. Sonali is a past recipient of the IBM Outstanding Technical Achievement Award, Women of Color STEM Technical All Star Award, and was recognized as an IBM developerWorks Professional Author in 2012.


developerWorks Professional author
        level

Anju Bansal (abansal@us.ibm.com), Senior Development Manager, IBM

Author photo for AnjuAnju Bansal is currently Senior manager of the IBM Big Data Accelerators group. Anju has been with IBM for more than 10 years and has led several development projects in the Enterprise Content Management and Information Server portfolio. In her current role, Anju’s team focuses on providing Big Data Analytic Accelerators that enable organizations to speed up deployments and solve big data problems sooner.



03 January 2013

Also available in Chinese Japanese

Machine data analysis is a business imperative

Half of Fortune 500 companies experience more than 80 hours of system down time annually. Spread evenly over a year, that amounts to approximately 13 minutes every day. While the outages may not happen every day, they may occur for about 1.5 hours after a week or 6 hours after a month and so forth.

As a consumer, the thought of online bank operations being inaccessible so frequently is disturbing. As a business owner, when systems go down, all processes come to a stop. Work in progress is destroyed and failure to meet SLAs and contractual obligations can result in expensive fees, adverse publicity, and loss of current and potential future customers. Ultimately the inability to provide a reliable and stable system results in loss of money. While the failure of these systems is inevitable, the ability to timely predict failures and intercept them before they occur is now a requirement.

A possible solution to the problem can be found in the huge volumes of diagnostic data generated at hardware, firmware, middleware, application, and storage and management layers that indicate failures or errors. Machine analysis and understanding of this data is becoming an important part of debugging, performance analysis, root-cause analysis, and business analysis.

In addition to preventing outages, machine data analysis can also provide insights for fraud detection, customer retention, and other important use cases.


Problem of machine data analysis is a big data problem

Machine data certainly fits the characteristics of big data.

Application architectures at each layer produce a variety of data outputs. The size of this data makes manual consumption very difficult. Sifting and correlating data across multiple files using pattern matching tools such as grep and awk is laborious and time consuming. The volume of this data is predicted to increase as more of the world is instrumented.

Figure 1. Variety, volume, and velocity in machine data
The variety, volume, velocity in machine data generated at all levels

IBM Accelerator for Machine Data Analytics

IBM Accelerators are software components that accelerate development and implementation of specific solutions or use cases on top of the big data platform.

IBM Accelerator for Machine Data Analytics is a set of BigInsights® applications that accelerate the implementation of use cases for machine data. These applications use BigInsights runtime technologies to support their implementation.

Figure 2. BigInsights and accelerators
Overveiw of BigInsights and Accelerators

Situation at a fictitious Sample Outdoors company

The afternoon of Saturday July 14th was not a good afternoon for the Sample Outdoors company. Websites were going down, customer calls were coming in complaining that purchases could not be made, and customers were unable to access their account information.

The Sample Outdoors company has many multi-tier applications, consisting of web front ends, application servers and database tiers.

Luckily, the Sample Outdoors company had implemented a machine data analytics solution with the help of IBM Accelerator for Machine Data Analytics. They have been saving machine data for many months now and are able to leverage it to understand a potential cause of such a situation.

On Saturday July 14th, the Sample Outdoors company was able to do the following.

  • Search for and acknowledge that there was an issue in their system.
  • Find patterns on historical data to determine prior issues with similar symptoms that had occurred previously.
  • Analyze the most influential causes for such situations to occur.

With this knowledge, they will put processes in place to better manage or even avoid such situations in the future.

Let's take a look at how the Sample Outdoors company implemented their machine data analysis solution using IBM Accelerator for Machine Data Analytics, and how their solution came in handy on that particular day.


Ten features to accelerate your machine data analysis

Take a look at the following overview and highlights of the features of IBM Accelerator for Machine Data Analytics that accelerate the implementation of machine data analytics solutions.

  1. To understand how to prepare the data with information useful for the analysis, refer to A little preparation before getting started.
  2. Refer to the Known log types section to learn how you can use known log types provided by the accelerator when machine data matches the known types supported by the accelerator.
  3. Refer to the Unknown log types section to use the generic log type for the unknown log types.
  4. Refer to the Bring in the data section to learn about bringing the prepared data into HDFS (Hadoop File System) so it is available for the extraction.
  5. To extract information or fields from text to prepare the data for analysis, refer to the Extract information from text section.
  6. Index all of the extracted information and get a consolidated view across all the data. For more information refer to the Shrink down, drill down, and acknowledge section.
  7. Observe chronological sequence of events leading to the error across logs. For more information, refer to the Chronological root cause analysis using search section.
  8. Refer to the Prepare for deeper analysis section to learn about building sessions combining relevant events together to prepare for deeper analysis.
  9. Would you like to identify frequently occurring events when errors occur? Get a peek into how to leverage the accelerator to help answer this question from the Identifying patterns section.
  10. Would you like to identify events that significantly influence errors? Get a peek into how to leverage the accelerator to help answer this question from the Understanding influencers section.

The Sample Outdoors company had the following applications representing customer management, with corresponding logs available.

  • CustomerFrontEnd application – An Apache web access-based application.
  • CustomerBackend application – An IBM WebSphere application server-based application.
  • CustomerDbApp application – An Oracle database application.

The following is the simplest case, but an overview of the accelerator is given.


A little preparation before getting started

Machine data or log data typically contains a series of events or records. Some records are as small as one line while others can span numerous lines. Typical examples of logs containing records that span multiple lines are application server logs which tend to contain XML snippets or exception traces, database or XML logs, or logs from any application that logs messages spanning across multiple lines. Apache web access logs or syslogs are good examples of logs containing records fitting in one line.

Often times, machine data is configured to omit information for brevity. Information such as server name, data center names, or any other concept applicable to a business can be complementary when used during the analysis of the data. You can associate this information in the metadata, to enrich further analysis.

First, you have to organize your data into directories or batches with similar content. Batches can contain individually gzipped files, uncompressed raw files, or sub-directories under them. Each subdirectory will be flattened into a batch. Metadata is associated for each batch provided as a json text file called metadata.json.

The MDAGenerateMeta.sh utility will generate the metadata file per batch based on your inputs. When you have fewer batches, you can bypass the utility and manually change an existing the metadata.json to match the content for your log data. While doing so, ensure that you provide a unique batchID, if you are creating a new batch.

MDAGenerateMeta.sh utility

A simple utility called MDAGenerateMeta.sh can be found in the bin directory of your accelerator's install location. This utility generates informational metadata in a file called metadata.json, based on inputs provided by the user. This utility associates a Metadata.json with every group or batch of logs, ensuring each Metadata.json contains a unique generated batchID. This metadata helps upcoming parsing, extraction and analysis steps. You can use templates to repeatedly run MDAGenerateMeta.sh with the same settings.

The following section discusses how to prepare known log types, as well as some lesser known ones.


The known log types

You can analyze Apache web access, WebSphere, syslogs, IBM Datapower or any character delimited file by simply using the corresponding log type during the preparation step, as shown in Table 1.

Table 1. Known log types supported out-of-the-box
Type of logLogType value to use in preparation
Apache web accesswebaccess
WebSpherewas
Syslogsyslog
Datapowerdatapower
Any character delimited filecsv
Anything elsegeneric

To extract interesting fields for the known log types, several rules are provided out-of-the-box. When additional fields need to be extracted, you can always customize. Application server-based logs can contain log entries from custom applications. Since, application developers are free to write their entries, some level of customization to get extract-specific information may be needed. Another example is when different flavors and versions of log types introduce variations in log content. These may require small amounts of customization.

You can always get started with a first round of analysis using the known types. Make your best guess for the log type. You can always fix it later! In this section, you see how the web access and WebSphere log types were used at the Sample Outdoors company. First, let's use the web access log type.

A web access log from the fictitious company Sample Outdoors customer front end application contains data as shown in Listing 1.

Listing 1. Sample apache web access log
|--------10--------20--------30--------40--------50--------60--------70--------80--------|
9.17.247.205 - - [14/Jul/2012:15:58:12 +0000] "GET /Sample Outdoors/us/customers.shtml
HTTP/1.1" 200 48490 "http://www.Sample Outdoors.com/Sample Outdoors/us/customer-info.html"
"Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR
2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; InfoPath.3; Tablet
PC 2.0)"8105 
9.17.247.205 - - [14/Jul/2012:15:58:21 +0000] "POST http://Sample Outdoors/us/customers-
order.php HTTP/1.1" 500 834 "http://www.Sample Outdoors.com/?p=732" "Mozilla/4.0
(compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 
3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; InfoPath.3; Tablet PC 2.0)"1250

Examine the log data shown Listing 1 and you will notice a pattern that includes an IP address followed by some characters and a timestamp, followed by the rest of the record. Identifying this pattern provides hints on how records are formed in this log data, and as a result, helps identify the primary timestamp. It is the primary timestamp because the log record could have other timestamps in the message, but they are not interesting when identifying record boundaries.

The following are a few key things to identify from the log.

  • The primary timestamp, one that defines the beginning of each record, is of the form 14/Jul/2012:15:58:12 +0000.
  • The primary timestamp is usually preceded by a string represented like 9.17.247.205 - - [.
  • A regular expression like "((\\n)|(\\r))+[^]]*\\[" can describe the repetitive occurrence of these characters at record boundaries.

The following are a few key things that are not in the log content, but are known to the Sample Outdoors application owner / data scientist, and can enrich the analysis.

  • The application name : customerFrontEnd.
  • The server name: webserver1.
  • The application falls under the overall category of customerManagement.

Next, use the information and associate it with the batches of data.

Listing 2 shows how MDAGenerateMeta.sh can be used to generate the metadata.json from the gathered information.

Listing 2. Script to generate metadata for customerFrontEnd application log
[biadmin@server bin]$ <MDA_Install_Location>>/bin/MDAGenerateMeta.sh
path="/opt/ibm/accelerators/input_MDA" logType="webaccess"
dateTimeFormat="dd/MMM/yyyy:HH:mm:ss Z" servername="webserver1"
application="customerFrontEnd" category="customerManagement"
preTimestampRegex="((\\n)|(\\r))+[^]]*\\["

Figure 3 shows the flow of how the metadata.json got associated with each batch or directory and its resulting content.

Figure 3. Using MDAGenerateMeta.sh utility to associate metadata with batches
Using MDAGenerateMeta.sh utility to associate metadata with batches

Click to see larger image

Figure 3. Using MDAGenerateMeta.sh utility to associate metadata with batches

Using MDAGenerateMeta.sh utility to associate metadata with batches

Next, let's use the WebSphere log type.

Listing 3 shows a WebSphere application log from the same fictitious company Sample Outdoors back-end application.

Listing 3. Sample WebSphere application server log
|--------10--------20--------30--------40--------50--------60--------70--------80--------|
[Sat July 14 03:58:12:906 PM] 00000048 InternalOracl I   DSRA8205I: JDBC driver name  : 
Oracle JDBC driver
[Sat July 14 03:58:19:296 PM] 00000047 WSChannelFram A   CHFW0019I: The Transport Channel 
Service has started chain HttpsOutboundChain:service.Sample Outdoors.org
[Sat July 14 03:58:19:312 PM] 00000047 ExceptionUtil E   CNTR0020E: EJB threw an 
unexpected (non-declared) exception

The following are a few key things to identify from this log.

  • The primary timestamp, one that defines the beginning of each record, is of the form Sat July 14 03:58:13 PM.
  • The primary timestamp is usually preceded by a string represented by "[". A regular expression like "((\\n)|(\\r))+\\[" can describe the repetitive occurrence of these characters at record boundaries.
  • The primary timestamp is missing a few values, namely, the year and the time zone. This article will provide the preferred year and time zone values to use.

The following are a few key things that are not in the log content, but are known to a Sample Outdoors data scientist, and can enrich the analysis.

  • The application name: customerApplication.
  • The server name: wasserver1.
  • The application falls under the overall category of customerManagement.

For any other log type, do the following.

  • Simply change the primary timestamp information.
  • Pick a log type, or use generic.
  • Change to a unique batchID, if you need it.
  • Add any other metadata, if you need it.
  • That's it!

Using the MDAGenerateMeta.sh script shown previously, or simply copying and changing an existing metadata.json file, create a metadata.json for the WebSphere log. The resulting metadata.json file used to associate with this log should look similar to what is shown in Listing 4. Notice that the missing value for year, namely 2012, and the missing value for timezone, namely +0000 is provided in the missingDateTimeDefaults field.

Listing 4. Generated metadata.json for the customerApplication
{logType:"was", batchId:"batch_was", serverName:"wasserver1",
application:"customerApplication",
category:"customerManagement",dateTimeFormat:"EEE MMM dd h:mm:ss:SSS
a",preTimestampRegex:"((\\n)|(\\r))+\\[",missingDateTimeDefaults:[{"year": "2012"},
{"timezone": "+0000"}]}

The unknown log types

What if the application server for the Sample Outdoors back-end application was Weblogic (or any other application server) and not IBM WebSphere? What about all the other layers of the application stack?

Of course, there are more types of logs than the known ones the accelerator provides. Whether the data is structured, semi-structured or unstructured, as long as it is time series-based textual data, it can be used for analysis. You can prepare batches of those logs the same way as shown previously. Simply make batches of similar data and associate metadata using the generic log type.

Commonly used rules for generic log types

  • Timestamps
  • Name value pairs
  • XML leaf and tag values
  • IPAddress
  • URL, URI
  • Severity
  • And many more!

Logs tend to contain several common fields and some log type specific fields. There are many scenarios that may not need those specific fields for analysis, or many not contain any. Several rules to extract commonly found interesting fields in logs are provided out of the box.

Numerous logs contain a bulk of the information in name value pairs. Others contain significant amounts of information in XML (leaf and tags), which will be extracted with the generic set of rules. To extract more fields, you can always customize. Part 2 in this series provides a hands-on tutorial to give a deep dive into customization.

Listing 5 shows an example of an Oracle database log, from a Sample Outdoors database application.

Listing 5. Oracle database log for the customerDbApp
<msg time='2012-07-14T10:58:16.723-0500' org_id='oracle' comp_id='rdbms'
type='UNKNOWN' level='16' version='1'>
<txt>*******************************************************</txt>
</msg>
<msg time='2012-07-14T10:58:18.723-0500' org_id='oracle' comp_id='rdbms'
type='UNKNOWN' level='16'>
<txt>Fatal NI connect error 12170</txt>
</msg>

The following are a few key things to identify from this log.

  • The primary timestamp, one that defines the beginning of each record, is of the form 2012-07-14T10:58:16.723-0500.
  • The primary timestamp is usually preceded by a string represented by "<msg time='". A regular expression like "((\\n)|(\\r))+&ltmsg time=\\'" can describe the repetitive occurrence of these characters at record boundaries.

The following are a few key things that are not in the log content, but are known to a Sample Outdoors data scientist, and can enrich the analysis.

  • The application name: customerDbApp.
  • The server name: dbserver1.
  • The application falls under the overall category of customerManagement.

Listing 6 shows the metadata.json for the Oracle customerDbApp.

Listing 6. metadata.json
{logType:"generic", batchId:"batch_oradb", serverName:"dbserver1",
application:"customerDbApp", category:"customerManagement",
dateTimeFormat:" yyyy-MM-dd'T'h:mm:ss.SSSZ", preTimestampRegex:"((\\n)|(\\r))+<msg 
time=\\'"}

Batches of logs from the three applications are now ready for analysis.


Bring in the data

Machine data resides on servers where they are generated. They need to be copied into the Hadoop File System (HDFS) for further analysis.

When you have a few batches, you can always upload the data to HDFS using the tools in the BigInsights console or Hadoop copy commands. When you have data in subdirectories, or have several batches to take care of at once, consider using the Import application.

Import application

The accelerator provides an Import application to copy previously prepared batches of logs into the Hadoop file system. Using this application, you can upload or replace one or more batches of similar or varied log types in one shot easily, as shown in Figure 4.

Figure 4. Use the Import application to import data into the Hadoop file system
Use the Import application to import data into the Hadoop file system

Click to see larger image

Figure 4. Use the Import application to import data into the Hadoop file system

Use the Import application to import data into the Hadoop file system

Application - Import: BigInsights technology used

  • Distributed copy application
  • Logs can be copied using ftp/sftp from a remote machine, or another location in hdfs.

Sometimes truncated data, or data with privacy information gets imported and needs to be replaced before it is analyzed. To replace previously imported batches, simply import the new content with a batchID in the metadata that is the same as the one to be replaced. To add a new batch, ensure that a new unique batchID is provided. Using the MDAGenerateMeta.sh utility during the preparation step will ensure this. If your data is in sub-directories, the application will flatten them into individual batches.


Extracting information from text

Prepared batches of text data can now be used to extract information.

Extraction application

The accelerator provides an Extraction application to take in previously prepared batches of logs and extract information from each of them. In addition to the metadata with each batch, the Extraction application can take additional configuration that commonly applies to multiple batches. Just like the Import application, the Extraction application can process one or more batches in one shot. New batches can be incrementally added by providing new batchIDs in the metadata for the batches that you want to add. To replace previously extracted batches, provide an existing batchID in the metadata that matches the ones to replace. The Extraction application prepares the data for cross log analysis and performs several steps along the way.

Application - Extraction: BigInsights technology used

  • Text analytics
  • JAQL - A JSON Query Language

Extraction application fixes all timestamps

When any information is missing logs, the Extraction application synthesizes the timestamp with the provided defaults in the metadata.json. In the interest of reducing the size of the log, common information is often omitted from the log. Certain fields in timestamps like time zones or year are common candidates for omission.

For an example, refer back to the CustomerApplication WebSphere logs shown in Listing 4.

The Extraction application also normalizes all timestamps to one single format. The primary timestamp formats in the three Sample Outdoors logs had different formats. They all need them to talk in one language to be able to analyze the logs together. Given the variety in machine data, this becomes a critical step for any further analysis.

Extraction application beefs up all records

In this step, the information provided by the application owner/data scientist in the metadata gets associated with each record. This prepares the data for cross log analysis using the metadata information along with fields extracted from the logs.

Extraction application prepares data for more interesting analysis

So far you looked at the original events, or records in the logs. You can optionally add a generalized version of the event to the record by masking certain fields.

Extraction tips

  • Start with small dataset.
  • Explore and validate extracted information.
  • Iteratively change metadata.json on a batch level or extract.config as needed.
  • Use resulting metadata and configurations for bigger data.

Extraction application produces sample results to view in sheets

As shown in Figure 5, you can get insight into the fields that got extracted, and validate the result produced by examining each column in the CSV output. Note that some columns may be empty because machine data is semi-structured or unstructured information. In such cases, more filtering on sheets may be needed to see non null values. Extraction prepares the data for the rest of the analysis. You can always export the full result as a CSV and perform further ad-hoc analysis in sheets.

Figure 5. Use the Extraction application to extract information
Use the Extraction application to extract information

Click to see larger image

Figure 5. Use the Extraction application to extract information

Use the Extraction application to extract information

Shrink down, drill down, and acknowledge

The next step of indexing processes the extracted information and prepares it for searching.

Index application

The Index application processes previously extracted batches of data. Just like the Import and Extraction applications, the Index application can process one or more batches in one shot. New batches can be added incrementally. Select the Re-index check box before running the application to delete the previously created index and create a new one.

Application - Indexing: BigInsights technology used

  • BigIndex
  • JAQL - A JSON Query Language

Once the index is created in HDFS, you can periodically copy it to the BigInsights console machine and make it available for searching.

CopyIndex.sh utility

Use the CopyIndex.sh utility found in the bin folder of the accelerator's install to periodically copy the index to the console machine. An example of using the copyIndex.sh utility is shown in Listing 7.

Listing 7. copyIndex utility
[biadmin@server bin]$ <MDA_Install_Location>/bin/copyIndex.sh -
hdfsIndexDir=hdfs://<server>:9000/<output_location_from_app>
copying indexes from hdfs..
Indexes successfully copied to local file system
MDA UI can be accessed at for secure install 'http://<hostname>:8080/datasearch/login.jsp'
MDA UI can be accessed at for non-secure install
'http://<hostname>:8080/datasearch/html/Search.html'.

The resulting indexed batches are now available for searching by using the appropriate URL, as shown previously in Listing 7.

The search user interface provides the following features.

Consolidated view of all the information across all the logs together

You can now view all the data across a timeline, which is available for searching via facets or a text search. All of the extracted fields from all the batches are available for searching. All of the metadata fields that the events are enriched with are also available as facets.

The index.config file is updated with the facet categories as new batches get indexed. You can control which facets to show/hide/rename from the search interface, or even remove from the index, by configuring this file, as shown in Figure 6.

Figure 6. Run Index application to prepare for search
Run Index application prepare for search

Click to see larger image

Figure 6. Run Index application to prepare for search

Run Index application prepare for search

Shrink down

You can shrink down the big data results to a time range or an event that is of interest to get a meaningful set of data. You can use the time range search or text search to accomplish this.

Drill down

You can then drill down into the facet values to find the events you are looking for. All of the metadata - applicationName, category, and servernames that the application owner added are also available as facets.

Acknowledge

Using combinations of shrink down and drill down, you can find, and hence acknowledge, the issues reported with evidence from the machine data to back it up.

The Sample Outdoors data scientists were able to acknowledge the error on the afternoon of Sat July 14th, as shown in Figure 7.

Figure 7. Shrink down on timeframe, drill down to find errors
Using the shrink down on timeframe, drill down to find errors

Click to see larger image

Figure 7. Shrink down on timeframe, drill down to find errors

Using the shrink down on timeframe, drill down to find errors
  • They were able to shrink down on Sat July 14th between the hours of 3pm and 4pm.
  • They were then able to drill down to the facet representing status codes, and acknowledge an error 500 in the web access logs from the customerFrontEndApp.
  • Did similar failures occur in the past? The Sample Outdoors data scientists zoomed out on the time range and noticed that numerous front end errors have occurred in the past.

Next, the data scientists were interested in understanding more about the events that caused the particular errors.


Chronological root cause analysis using search

Using the search user interface, you can now look at events in chronological order across all logs and gain more insights into the cause of the error, as shown in Figure 8.

Figure 8. Chronological root cause analysis across logs
Figure showing the chronological root cause analysis across logs

Click to see larger image

Figure 8. Chronological root cause analysis across logs

Figure showing the chronological root cause analysis across logs

The Sample Outdoors data scientists were able to acknowledge the error on the afternoon of Sat July 14th, including the following.

  • The consolidated view of all logs showed that an error in the web access application was preceded by errors in the WebSphere application which was, itself, preceded by an error in the Oracle database application.
  • A simple sequence of events indicated that in this case, an error in the back-end database caused the error.
  • They also noticed some problems at start up, and would like to understand any relation those may have with further deeper analysis. Refer to the first event shown previously in Figure 8, referring to message represented by WSVR0002I on WebSphere. You can look out for this error and its relation to errors in web access logs during deeper analysis.

The appropriate application owners were contacted to investigate the particular issue on Sat July 14th. Next, the data scientists needed to understand the causes of such failures in their applications in order to reduce future disruption.


Prepare for deeper analysis

Log records or events individually may not contain enough information about the cause of an error. Combining them with other relevant events can make deeper analysis possible.

The accelerator provides two applications to create groups of relevant events or sessions.

TimeWindow application

Build sessions with one log type, combining events based on time gaps or time windows.

Application - TimeWindow App and JoinTransformation App: BigInsights technology used

  • JAQL - A JSON Query Language

Use the time gap setting in the timegapTransformation.config file for logs that tend to show gaps in activity. Typical examples are web access logs which capture user activity for some time, followed by gaps in activity. You can form sessions for logs which have a more continuous stream of activity by using the time window flavor of TimeWindow application.

JoinTransform application

Build sessions combining two different log types, including relevant layers from application architecture using the JoinTransformation.

Transformation tips

  • Start with small dataset.
  • Change the timegap.config or joinTransformation.config to build relevant sessions.
  • Use configurations relevant to the data, and use case for bigger data.

You can also use the JoinTransformation application to build sessions from a single log type.

For both of the applications, configurations control the transformations. You can tweak configurations to make the analysis more meaningful for your data and use case. All of the fields extracted from the logs, or enriched via the metadata, can be used in transformations via the config files. You can export a sample from the resulting sessions into a CSV file for visualization, and this setting is available in the config also.

Use JoinTransformation configuration

The Sample Outdoors data scientist wanted to analyze across the web application layer and the WebSphere application layer. They wanted to identify patterns in the WebSphere application when an error occurred in the web access layer. They also wanted to understand which events occurring within a certain time window in the WebSphere application layer significantly influence errors in the web access layer.

They used the JoinTransformation configuration as shown in Listing 7 and shown in Figure 5. The configuration allowed them to do the following, as shown in Listing 8.

Listing 8. JoinTransform config and lookup table
[
{
"seed_log_type": "webaccess",
"content_log_type": "was",
"device_id_field_names_seed": "category",
"device_id_field_names_content": "category",
"lookup_table_dir": 
"/_default_match_all_join_metadata_table",
"message_type_field_name_seed": "CodesAndValues",
"match_type_seed": "Prefix",
"match_type_content": "Prefix",
"message_type_field_name_content": "MethodidWS",
}
]
                
Lookup table
fact_field,context_field,time_window
3,"", 20000
4,"", 20000
5,"", 20000

Join web access logs with WebSphere logs.

  • Seed log type – webaccess
  • Content log type - was

Provide join keys for each log. They used the category field that was earlier added as metadata.

  • Device_id_field_names_seed – category
  • Device_id_field_names_content – category

Filter the joins to only include errors in the webaccess log. They accomplished this by specifying filters on the values for the CodesAndValues field, which represents http status code.

  • Message_type_field_name_seed – CodesAndValues.
  • Match_type_seed – Prefix.
  • Use a lookup table to filter the values for CodesAndValues to be included in the join. Values starting with 3,4 or 5 were included. Note that the match type of prefix allows such a match.
  • Specify the time window to look back into - Lookup table time_window – 20000 (ms).

They wanted to look for patterns in messages. A field that represents message IDs is used.

  • Message_type_field_name_content – MethodidWS

Figure 9 shows the join logs in preparation for analysis.

Figure 9. Join logs
Figure 9 shows the join logs in preparation for analysis.

Click to see larger image

Figure 9. Join logs

Figure 9 shows the join logs in preparation for analysis.

Next, the Sample Outdoors data scientists were ready to start deeper analysis.


Identifying patterns

It is often desirable to identify what events occur around a particular event or error.

Frequent Sequence Analysis application

The accelerator provides a Frequent Sequence Analysis application that operates on previously prepared sessions, and provides insights into the most frequent sequences of events across sessions.

Application - Frequent Sequence Analysis application: BigInsights technology used

  • System ML
  • JAQL - A JSON Query Language

As before, configurations control the analysis. You can tweak configurations to make the analysis more meaningful for your data and use case.

Any extracted field that is also part of the session can be used in Frequent Sequence Analysis. Results are available in CSV format. You can use filtering in sheets to find patterns that contain values of interest. Results are available as a chart and can be seen in a dashboard.

The data scientists used the FrequentSequence analysis configuration as shown in Listing 8 which allowed them to find patterns of messagesIDs (which is represented by the field methodidWS), as shown in Listing 9.

Listing 9. Frequent Sequence analysis config
"eventField": "MethodidWS"

The Sample Outdoors data scientists were able to notice a pattern of events, using their Frequent Sequence Analysis dashboard, as shown in Figure 10.

Figure 10. Frequently occurring sequences in WebSphere log before errors surface in web access logs
Frequently occurring sequences in WebSphere log before errors surface in web access logs

Click to see larger image

Figure 10. Frequently occurring sequences in WebSphere log before errors surface in web access logs

Frequently occurring sequences in WebSphere log before errors surface in web access logs

Look again at the first event shown in Figure 8 that refers to messageID WSVR0002I on WebSphere. You can look out for this error and its relation to the 500 error in web access logs during deeper analysis.

The Sample Outdoors scientists were able to confirm the potential relation they had observed during the chronological root cause analysis experience.

They now also have other frequently occurring patterns to understand from a root cause analysis perspective.


Understanding influencers

It is also often desirable to get insight into which events influence the cause of an error.

Significance Analysis application

The accelerator provides a Significance Analysis application that operates on previously prepared sessions, and provides insights into events influencing other events overall. The results can be filtered down to identify influences on a certain error.

Application - Significant Analysis application: BigInsights technology used

  • Machine learning
  • JAQL

Note that only sessions resulting from the JoinTransformation application can participate in Significance Analysis.

Similar to the Frequent Sequence Analysis application, configurations control the analysis, and you can tweak configurations to make the analysis more meaningful for your data and use case.

The following are some tips on terminology.

  • Any field from the seed part of the session can be used as labels.
  • Any fields from the content part of the session can be used as features influencing labels.
  • Refer to Figure 5 for examples of seed and content fields when forming session using the JoinTransformation application.
  • Results are available in CSV format. You can use filtering in sheets influencers for a specific value.

The data scientists at Sample Outdoors now used the sessions generated to understand which events in the WebSphere application layer significantly influence errors in the web access layer.

They provided a significance analysis configuration as listed in Listing 8. They picked CodesAndValues from the seed part of the session as the label field, and MethodidWS from the content part of the session as the feature, as shown in Listing 10.

Listing 10. Significance analysis configuration
[
{
"label_field": "CodesAndValues",
"features": "MethodidWS"
}
]

Figure 11 shows the results of the significance analysis with the configuration shown previously in Listing 10.

Figure 11. Overall view of errors in WebSphere influencing errors in web access layer
Overall view of errors in WebSphere influencing errors in webaccess layer

The data scientists at the Sample Outdoors company identified that the following errors in WebSphere were most influential in the cause of an error in the web access layer, in the order listed.

  • CNTR0020E EJB threw an unexpected exception during invocation of method.
  • WSVR0002I Server open for e-business, problems occurred during start up.

From the Frequent Sequence Analysis, they had also identified that WSVR0002I frequently preceded CNTR0020E. Refer back to Figure 10 for an example.

Continuing with these insights, they were empowered to perform further analysis into the deeper levels of their application stack.


Accelerate the implementation of your solutions

The accelerator also ships chained applications to expedite potentially more frequent combinations of tasks. These chains include the following.

  1. Import-Extraction
  2. TimeWindow Transformation-Frequent Sequence Analysis
  3. JoinTransformation- Significance Analysis

Once the analysis to be performed is identified, build your own chains to simplify execution.

At the Sample Outdoors company, in addition to using the chains, new ones were built to execute JoinTransformation – FrequentSequence Analysis. Deeper analysis continued into other layers of the application stack to link the WebSphere application with the back-end Oracle database application.


Conclusion

With the help of a situation at the fictitious Sample Outdoors company, you looked at how IBM Accelerator for Machine Data Analytics can accelerate the implementation of a solution across various logs. You saw how the various applications that are shipped with the accelerator can be individually used to accomplish this task.


About this article series

One of the primary advantages and strengths of IBM Accelerator for Machine Data Analytics is the capability and ease with which the tools can be configured and customized. This series of articles is for those who want to get an introduction to the accelerator and further accelerate the analysis of machine data with the idea of getting custom insights.


Acknowledgements

We would like to acknowledge Tom Chen (thchen@ca.ibm.com) for his technical review. We would also like to acknowledge the entire Machine Data Accelerator team for their contributions to this product.

Resources

Learn

Get products and technologies

  • Build your next development project with IBM trial software, available for download directly from developerWorks.
  • Evaluate IBM products in the way that suits you best: Download a product trial, try a product online, use a product in a cloud environment, or spend a few hours in the SOA Sandbox learning how to implement Service Oriented Architecture efficiently.

Discuss

  • Get involved in the My developerWorks community. Connect with other developerWorks users while exploring the developer-driven blogs, forums, groups, and wikis.

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Big data and analytics on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Big data and analytics, Information Management
ArticleID=853449
ArticleTitle=IBM Accelerator for Machine Data Analytics, Part 1: Speeding up machine data analysis
publish-date=01032013