Custom Annotation Query Language (AQL) rules

You can define custom rules for splitting and annotating log records in AQL.

AQL is similar to Structured Query Language (SQL) where data generated by executing AQL statements is stored in tuples. A collection of tuples that are generated for a statement forms a view, which is the basic AQL data model. All tuples for a view must have the same schema.

AQL is a feature of the IBM® InfoSphere® BigInsights® platform. For more information, see http://www-01.ibm.com/support/knowledgecenter/SSPT3X_3.0.0/com.ibm.swg.im.infosphere.biginsights.aqlref.doc/doc/aql-overview.html?lang=en.

You must be aware of the key concepts of AQL. Some of the key concepts are as follows:
  • You can add custom annotation logic in two ways. You can add custom .aql files or precompiled AQL modules, which are stored in .tam files, to the rule set directory. For more information, see Creating Rule Sets.
  • If you want to use a custom AQL script, you must add a .aql extension to any file that contains AQL statements. You can group related AQL files in the same directory on a file system. The directory then becomes an AQL module. Declare the module at the beginning of each .aql file. Then, when you want to reuse the same logic elsewhere, you can import the modules into other AQL files that are in a different directory.
  • The text that is sent to the AQL engine in IBM Operations Analytics Log Analysis for annotation is represented in a specific view that is called Document. The Document view is populated by the engine when it runs. Each AQL statement can access this view and perform operations on it.
  • The fields in an AQL tuple must belong to one of the built-in scalar types. The types are Boolean, Float, Integer, List, Span, String, and Text.
  • The Span type represents a contiguous region of text in a text object that is identified by the beginning and ending positions. For examples, see Custom annotations and splitters.
  • The following are some of the primary AQL language statements:
    • import, export, and module are used to create, share, and use modules
    • create table is used to define static lookup tables to augment annotations with additional information
    • create dictionary is used to define dictionaries that contain words or phrases. The dictionary is used to identify matching terms across input text through extract statements or predicate functions.
    • create view is used to create a view and to define the tuples inside that view
    • create external view is used to specify additional metadata about a document as a new view. You can use this view alongside the predefined Document view that holds the textual and label content.
    • extract is used to extract useful data from text.
    • select is used to provide a powerful mechanism for constructing and combining sets of tuples that are based on various specifications
  • AQL also has the following built in functions that you can use in extraction rules:
    • Predicate functions such as Contains, Equals, and Follows.
    • Scalar functions such as GetLength, GetString, and LeftContext.
    • Aggregate functions such as Avg, Count, Min, and Max.
  • You can also add user-defined functions (UDFs) that you define to AQL. For more information, see http://www-01.ibm.com/support/knowledgecenter/SSPT3X_3.0.0/com.ibm.swg.im.infosphere.biginsights.aqlref.doc/doc/udfs.html?lang=en.

For examples of AQL statements, see the AQL files that are provided with each of the Insight Packs that are installed with IBM Operations Analytics Log Analysis. ThreadID.aql contains the views for annotating the thread Id field from a WebSphere® log file. The ThreadID.aql file is in the <HOME>/unity_content/WAS/WASInsightPack_v1.1.0/extractors/ruleset/annotatorCommon directory.

Requirements for a custom splitter in AQL

If you define your own splitter in AQL, you must name the AQL view LogRecord.

You also must define the columns in the AQL view and the corresponding data types as outlined in the following table.
Table 1. LogRecord columns and data types
Column Data type Description
logSpan Span

The span of the input document that this log record represents.
logText String

The text of the log record.
timestamp String

The time stamp, if there is any, that is associated with the log record. If the log record does not contain a time stamp, this entry contains an empty string.
type String

A single character that denotes the type of the log record. The value for this entry is A, B or C. For more detailed information about these values, see Custom annotations and splitters.

Tooling for custom AQL rules

You use the Eclipse based tools that are provided by the IBM InfoSphere BigInsights platform to help you to develop and test AQL rules. You can use the tools to import sample log file data, write AQLstatements that extract the relevant information, and to test the AQL statements before you install your custom Insight Pack on the IBM Operations Analytics Log Analysis server.

For more information about how to install the tools, see Tools for extending IBM Operations Analytics Log Analysis.

Best practices

To help ensure that you write effective and reusable rules, read the best practices section of the documentation before you create your own AQL rules for IBM Operations Analytics Log Analysis. For more information, see Best practices information.