Understand the Extraction application
For an overview of the Extraction application, refer to the "Extracting information from text" section from Part 1: Speeding up machine data analysis of this series.
Perform the following steps.
- The Extraction application has a set of rules for a few known log
types, and a generic set of rules for the lesser known log types.
Expand the AQL folder to see the files containing
these rules and how they are organized. Figure 5 shows how the AQL
rules are organized for each of the log types.
Figure 5. Peek into the Extract application
Notice the naming convention used. Each log type has a corresponding extractor_logtype.aql file. This is mandatory to follow when you build the email logtype.
Figure 5 also shows the extractor_logtype.aql files for the out-of-the-box log types. The extractor_logtype.aql is a top-level module that includes all of the rules for that log type. Typically, all of the rules for the log type are defined in a subdirectory.
- Take a look at the AQL rules under the common
directory, which represent the generic set of rules. Expand the
common directory and review the AQL rules, as
shown in Figure 6.
Figure 6. AQL rules included in the generic type
Similarly, you can take a look at the rules included for the known types.
- The AQL rules are called by a custom JAQL module. To look at the
compiled AQL rules included in the JAQL module, expand the
src/jaql/custom_modules folder. Figure 7
shows the compiled rules that are exposed via the custom module.
Figure 7. Compiled AQL rules exposed in the custom module
- Notice the resulting naming convention. Each log type has a corresponding extractor_logtype.tam file in the custom module/extractor folder. This is helpful to note when you build the email log type. The Extraction application contains several other JAQL scripts and Java UDFs, but they are not of interest when building a new log type.
By changing these AQLs, you can always customize any of the existing rules for the existing known and generic log types, or include any new rules for the existing known and generic log types.