Eclipse Test & Performance Tools Platform, Part 2: Monitor applications

Producing the parser

The sensor reads data. The extractor subdivides the data into records. The role of the parser is to extract specific fields from each record and use those values to construct a complete CBE XML record.

The role of the parser

The parser may extract some fields from the log file directly, such as a time stamp, host name, daemon name, and a text message. The parser may also infer data from a record. For example, the parser may detect that the record originated with a software service and set the CBE componentIdType attribute to ServiceName. In other instances, the parser may add data to a record. In particular, if a log entry doesn't record the day, month, year, time, and time zone of the event, the parser must add that data to create a valid CBE.

To put the parser for the daemon.log example in perspective, Listing 15 shows a valid CBE XML record for the log entry Mar 2 06:27:35 db popa3d[7964]: Session from 71.65.224.25. Some of the attributes are plainly derived from the original log entry; others will be manufactured from implied data. (Many of the values of the attributes come from the Common Base Events Specification. It's helpful to use that document while creating your parsers.)

                    
<CommonBaseEvent 
    creationTime="2006-03-02T13:27:35.000Z" 
    globalInstanceId="A1DAABECA2ACB4F0E8E9E8C475042F1B" 
    msg="Session from 71.65.224.25" 
    version="1.0.1">
  <sourceComponentId 
      component="popa3d" 
      componentIdType="ServiceName" 
      location="db.linux-mag.com" 
      locationType="Hostname" 
      subComponent="7964" 
      componentType="daemon"
   />
  <situation 
      categoryName="StartSituation">
      <situationType 
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
        xsi:type="StartSituation" 
        reasoningScope="EXTERNAL" 
        successDisposition="SUCCESSFUL" 
        situationQualifier="START INITIATED"/>
  </situation>
</CommonBaseEvent>

Also keep in mind that (at a minimum) every CBE must define the creationTime attribute, the msg attribute, and the sourceComponentId element, which in turn must have the six attributes shown in Listing 15. The situation element (among others) is optional, but is part of the example to elaborate upon the event.

Back to top

How the parser works

Click Parser in the Generic Log Adapter perspective to begin the process of defining it. Figure 11 shows what the completed parser looks like. There is one parser task for every attribute and element in the CBE shown in Listing 15.

The parser works in two phases. First, it divides the incoming record (from the extractor) into positions, or numbered parts, in which each part is separated from the other by the separator token. If no separator token is specified, this step is skipped. Then the parser divides the record into designations, or (name, value) pairs, in which each (name, value) pair is two strings joined by the designation token. If no designation token is specified, the latter step is skipped.

Consider this example: If the separator token is the regular expression [ ]+, the designation token is = (equal sign), and the parser is handed the record:

03/05/06 12:51:06EST Mail name=joe action=login authentication=password

the parser would define six positions and three designations, as shown in Table 1.

Note: If your incoming record begins with the separator token, position 1 is created, but left empty.

You can use all the defined positions and designations to simplify each parser task. For instance, to create the creationTime attribute, you need only parse position 2. Of course, the entire original record is always available. However, positions and designations make each parsing task faster and easier to manage because the source string is smaller. In many cases, you can use a position or designation directly for a CBE value.

Back to top

Parse the sample log entries

Click Parser again. For convenience, break each daemon.log entry into two positions using the separator token :[ ]+ (a colon followed by one or more spaces). The daemon.log log entries don't have (name, value) pairs, so the designator token is omitted. These settings are shown in Figure 12. Now, save your work.

Set the creationTime

Set the first required field in the CBE: creationTime. The goal is to transform the time stamp provided with the daemon.log record into a time format compatible with the XML schema dateTime data type. As a convenience, the adapter can automatically permute a time format understood by class java.text.SimpleDateFormat into the XML schema data type.

To set the creationTime field, complete these steps:

1. Expand the parser and select creationTime. This is a required CBE attribute, so select the Required by parent check box.
2. Click the substitution rule associated with creationTime.
3. For Positions, type 1 because position 1 contains the time stamp to extract.
4. For Match, provide the regular expression ^(\w{3})\s+(\d{1,2})\s+([\d:]+)\s+.*$. This expression captures the month name as $1, the day of the month as $2, and the time of day as $3.
5. For Substitute, supply $1 $2 @YEAR $3 @TIMEZONE.
   Substitute is used instead of the entire incoming record in the rest of this specific parsing task. $1, $2, and $3 came from the previous step. However, because the time stamp doesn't include a year or a time zone, the year and time zone associated with the current context instance, represented by the shorthand @YEAR and @TIMEZONE, respectively, are used instead. Therefore, for the first daemon.log record, the settings in Substitute yield the string Mar 02 2006 06:27:35 -0700.
6. Ignoring the Substitute extension class field, which allows you to provide a Java class to do additional substitutions, transform the result of the substitution to the right type. You can use a java.text.SimpleDateFormat format string to do the heavy lifting. Set Time format to MMM dd yyyy hh:mm:ss Z, indicating a three-letter name of the month; a two-digit day of the month; a four-digit year; hours, minutes, and seconds separated by colons; and an RFC 822 time zone.

Figure 13 shows the final settings for creationTime. If you save the configuration file and rerun the adapter, the Formatter Result pane should show a new XML record with attribute creationTime="2006-03-02T13:27:35.000Z".

Getting the message

The msg attribute is another required CBE attribute. Add this attribute and create the parser task to extract a suitable value:

1. Right-click CommonBaseEvent, then click Add > msg.
2. Click msg, then select the Required by parent check box.
3. Expand msg, then click Substitution Rule.
4. Specify 2 in the Positions field because the message portion of the log entry is located in position 2. (It's everything after the separator token.)
5. For Match, specify a regular expression that selects the entire string. The regular expression ^(.*)$ captures everything in $1.
6. For Substitute, specify $1.

Figure 14 shows the final settings.

Save the configuration file and click Rerun adapter, found in the Extractor Result pane. Click Next event and switch to the Formatter Result pane. You should see a new msg attribute that looks like msg="Session from 71.65.224.25".

Find the source

The last mandatory part of a CBE record is the sourceComponentId, used to record the component (service, system, and so on) that's affected by the event. In the instance of daemon.log, the components affected are software services running on a specific host. The parser's job is to capture and record the specifics.

Right-click CommonBaseEvent once again, and then click Add > sourceComponentId. (Figure 15 shows all the possible attributes and elements you can add to a CBE.) For brevity, Table 2 shows all the settings required for sourceComponentId. One new setting is Default value. If a match is made by a parsing rule, but no substitute value is provided, the Default value is used.

If you make all the changes listed in Table 2, and save and rerun the adapter, you should yield CBE event records that resemble Listing 15. As an additional exercise, add a situation to the CBE. Situations categorize the type of situation that initiated the event. For instance, you might create a parser to create a StartSituation whenever the daemon is initially contacted for service or create another parser to create a RequestSituation when a request is made.

Situations aren't required (hence, Required by parent can be disabled), but you may find them useful to add granularity to your CBE records. If you create a situation and add a series of possible situation parsers, select the Child choice check box if processing can stop after the first match is made.

Here's a helpful tip for debugging your parsers: If a property is required, but not found in the incoming record (passed to the parser from the extractor), the Formatter Result pane for that record will be empty. In other words, required properties behave like logical AND: If one match fails, processing for that record stops. It's often useful to clear the Required by parent check box to debug rules. Build your rules slowly and incrementally, and watch the Problem pane for clues.

7 of 12 | Previous | Next

Comments

Back to top

Table of contents

• Before you start
• Transforming and analyzing log files
• Installing the prerequisite software and components
• Creating an adapter
• Specifying the sensor
• Editing the extractor
• Producing the parser
• The formatter and organizing the outputter
• Running the GLA
• Summary
• Resources
• About the author

---

7 of 12 | Previous | Next

Dig deeper into Open source on developerWorks

IBM SmartCloud trial. No charge.

Unleash the power of hybrid cloud computing today!

---

Public and private solutions in one trial.

Special offers