Format file specifications
The format file describes the patterns that the agent looks for to match events in the monitored logs. The format file consists of one or more format specifications.
You can change the format file while an agent instance is running. The file is read by the agent when it starts, and is monitored for changes to its time stamp every 60 seconds thereafter. If the time stamp of the file changes, the agent reinitializes its configuration dynamically, without requiring a restart. For more information, see Changing the agent configuration and format files.
To create new patterns to match an event, use the new regular expression syntax that consists of the following parts:
- Format header
- Regular expression
- Slot mappings
- End statement
The format header contains the REGEX keyword, which informs the agent that you are using a regular expression to match the pattern in the monitored log.
REGEX REExample
*DISCARD*
as your event class, any
log records matching the associated pattern are discarded, and no events are generated for them. For
example: REGEX *DISCARD*
When a pattern is matched, nothing is written to the
unmatch log. The log file status records that are matched include these discarded events.After the format header, the format content consists of a regular expression on the first line, followed by mappings. Each mapping is shown on a separate line and these mappings are described in the following example.
All lines that match the regular expressions are selected and sent to the monitoring server as events. The regular expression contains subexpressions. You can use the subexpressions to match specific parts of these lines that are the same to a variable called a slot in the Event Integration Facility.
Error: disk failure
Error: out of memory
WARNING: incorrect login
Error
and ignore the line that begins with Warning
. The regular
expression must match the lines that begin with Error
and also include a
subexpression. The subexpression is denoted by parentheses and it must match only the input text
that you want to assign to the msg slot. The following format definition
is a simple regular expression with only one
subexpression:REGEX REExample
Error: (.*)
msg $1
END
Based on this format specification, and the preceding set of log data, the agent generates two
events. Both events are assigned the REEXample
event class. In the first event, the
disk failure
value is assigned to the msg slot. Also, in
the second event, the out of memory value is assigned to the
msg slot. Because the Warning
line did not match the
regular expression, it is ignored and no event is generated.
When you assign the value of $1 to the msg slot, you assign it the value of the first subexpression.
Error: disk failure on device /dev/sd0: bad sector
Error: disk failure on device /dev/sd1: temperature out of range
You
can include a description of the disk on which the error occurred, and more specifically the disk
error in the event.REGEX DiskFailure
Error: disk failure on device (/dev/sd[0-9]):(.*)
device $1 CustomSlot1
msg $2
END
You assign these two subexpressions to event slots. The two events that are generated contain the following values:
"device=/dev/sd0" and "msg=bad sector"
"device=/dev/sd1" and "msg=temperature out of range"
If you use EIF to generate the first event, it displays as shown in the following example:
DiskError;device='/dev/sd0';msg='bad sector';END
If the event is sent to the Cloud APM server, the slot that is named msg is assigned to the Performance Management agent attribute with the same name. But the device slot has no predefined attribute.
If you need to see the value that is assigned to device directly on the Cloud APM console, or write thresholds against it, you must assign it to a Performance Management attribute.
- Ten string type attributes that range from CustomSlot1 to CustomSlot10
- Three integer type attributes that range from CustomInteger1 to CustomInteger3
CustomSlot
and CustomInteger
attribute names are
case-sensitive, so you must enter the names exactly as shown.You assign a slot from the event definition to one of these custom Performance Management attributes in the format file.
REGEX DiskFailure
Error: disk failure on device (/dev/sd[0-9]):(.*)
device $1 CustomSlot1
msg $2
END
When the event is displayed in the Application Performance Dashboard,
the value that is assigned to the device slot is assigned to the
Performance Management CustomSlot1
attribute. You view this value in the Cloud APM console or use it to define thresholds. You can assign
any slot in the event definition to any of the 10 custom agent attributes in the same manner, by
using "CustomSlotn
", where n is a number from
1 - 10, next to the slot definition.
In this example, the first subexpression is defined specifically as
(/dev/sd[0-9])
, but the second subexpression is defined generally as
(.*)
. In defining the regular expression as specifically as possible, you improve
performance. Therefore, if you enter a search for an error on a device that does not match the
specific error message that is defined here, the search procedure stops immediately when the error
is not found. Time is not wasted looking for a match.
REGEX REExample
Error:
msg $1
END <EOL>
<EOF>
CustomInteger1 to
CustomInteger3 are 64-bit custom integer attributes. You can use them in
the same manner as the string type CustomSlot
attributes. You can use these
attributes to map individual slots, or subexpressions, from the log file to individual Cloud
APM attributes. Because these attributes are
numeric, you can use arithmetic comparisons on them, such as < and
>, which is not possible with the string attributes.
Oct 24 11:05:10 jimmy fschecker[2165]: Filesystem /usr is 97% full.
REGEX FileSystemUsage
^([A-Z][a-z]{2}) ([ 0-9][0-9]) ([0-9]{2}:[0-9]{2}:[0-9]{2}) (.*?) (.*?):
Filesystem (.*?) is ([0-9]+)% full\.$
Month $1 CustomSlot1
Date $2 CustomSlot2
Time $3 CustomSlot3
Host $4 CustomSlot4
Service $5 CustomSlot5
Filesystem $6 CustomSlot6
PctFull $7 CustomInteger1
msg PRINTF("%s: %s% full", Filesystem, PctFull)
END
^
and $
symbols on the second
and third lines must be on a single line.( Class == 'FileSystemUsage' AND CustomInteger1 >= 95)
A different event can then use CustomInteger1 for a different purpose and not trigger this threshold accidentally. In summary, you can now write a threshold in Performance Management that uses arithmetic operators on the CustomInteger
attributes, which is not possible with the CustomSlots
attributes.
CustomInteger
attributes, the resulting
value might be zero or some unexpected value.