Date-time annotation formats

The date-time (timestamp) annotation formats are specified in the DateTime-consolidation_BI.aql file.

View: DateTimeOutput

File name: DateTime-consolidation_BI.aql

The following date-time annotation formats are available.
Table 1. Date-time annotation formats
Format name Pattern Examples
DT1 A timestamp in the format DT, where:
  • D is the D1, D2, D3, D4, D5, D6, or D7 date format
  • T is the T1, T2, T3, T4, T5, or T6 time format
 Sunday, the 3rd of December, 2012 23:11:59.120 IST

3/12/2012 2:11:59+05:30

6/10/12 2:48:28:381 MDT

Thu Nov 22 06:28:48 EST 2012
DT2 A timestamp in the format D4:T3. 3/December/2012:2:00:00.000

1st/Dec/2012:23:11:59+05:30
DT3 A timestamp in the format D4:Hour:Minute:Seconds Z, where Z is an RFC 822-four-digit time zone format that conforms with the Java™ SimpleDateFormat class. 3/December/2012:02:00:00 -0030

1st/Dec/2012:23:11:59 +0530

Other formats

The view DateTimeOutput also supports timestamp formats in these ISO date formats:
  • yyyy-MM-ddTHH:mm:ss.SSSZ. For example, 2013-02-27T13:57:21.836+0000
  • yyyy.MM.ddTHH:mm:ss.SSSZ. For example, 2013.02.27T13:57:21.836+0000
Variations of these formats are also supported:
  • yyyy/MM/dd-HH:mm:ss.SSSZ. For example, 2013/02/27-13:57:21.123+0000
  • yyyy/MM/dd-HH:mm:ss.SSS. For example, 2013/02/27-13:57:21.123
  • yyyy-MM-dd-HH:mm:ss. For example, 2013-02-27-13:57:21

Date-time stamps with no year value

Some applications write logs with no year in the date/time stamp. For example, the UNIX messages log, such as shown here:

Apr 24 09:41:16 bluewashmachine symcfgd: subscriber 2 has left -- closed
 0 remaining handles

Apr 24 09:41:20 bluewashmachine rtvscand: New virus definition file loaded.
 Version: 150423c.

Apr 24 09:41:38 bluewashmachine kernel: type=1400 audit(1366792898.697:52164):
 avc:  denied
  { module_request } for  pid=18827 comm="smtpd" kmod="net-pf-10"
 scontext=system_u:system_r:postfix_smtpd_t:s0
 tcontext=system_u:system_r:kernel_t:s0 tclass=system

Apr 24 09:41:38 bluewashmachine kernel: type=1400 audit(1366792898.822:52165):
 avc:  denied
  { module_request } for  pid=18833 comm="proxymap" kmod="net-pf-10"
 scontext=system_u:system_r:postfix_master_t:s0
 tcontext=system_u:system_r:kernel_t:s0 tclass=system

In this case, the Generic-dateTime-Split splitter identifies the string Apr 24 09:38:58 as a valid date-time stamp. To meet the required date formats of IBM® Operations Analytics, a valid year must be associated with the date-time string. The generic-dateTime-split splitter address this problem by placing a yyyy value at the beginning of the identified date-time stamp format. As a result, the timestamp now reads 2013 Apr 24 09:38:58.

You must update the timestamp format for files this type in the indexConfig. For example, if you want IBM Operations Analytics to ingest log records with the timestamp format MMM dd HH:mm:ss, the dateFormat must be specified as shown here. 
 "timestamp": {
            "searchable": true,
            "filterable": true,
            "retrievable": true,
            "dataType": "DATE",
            "tokenizer": "literal",
            "sortable": true,
            "source": {
                "dateFormats": [
                    "yyyy MMM dd HH:mm:ss"
                ],
                "paths": [
                    "metadata.timestamp"
                ],
                "combine": "FIRST"
            },
            "retrieveByDefault": true
        },
The supported date formats without a year are:
  • Apr 16 (MMM dd)
  • 16 Apr (dd MMM)
  • 16 April (dd MMM)
  • April 16 (MMM dd)
Year end scenario: The generic-dateTime-split splitter applies the current year to any timestamp that is ingested where no year can be discerned from the log record. The exception is when the log record is ingested where current system time of the IBM Operations Analytics server identifies the month as January, but the incoming date/timestamp is December. In such situations, the year value that is applied is the current year minus 1.

Logs with no date

Some data sources that are ingested by IBM Operations Analytics do not support a date within the timestamp. For example, some lines of a log display the characteristic shown here in bold:
00:11:35.103 INFO  [main] - Server accepting connections on rmi://9.11.222.333:1099/

09:34:33.071 INFO  [main] - Server accepting connections on tcp://9.11.222.333:3035/

To ingest such a data source, IBM Operations Analytics provides a splitter rule set Generic-timeOnly-Split, which you can use along with the Generic-Annotator to ingest such a log. The splitter prepends a date to the identified time from each record of the log.

You must update the timestamp format for files of this type in the indexConfig. For example, in order for IBM Operations Analytics to ingest a log with records with timestamp such as 09:34:33.071, the dateFormat must be specified as here. 

 "timestamp": {
            "searchable": true,
            "filterable": true,
            "retrievable": true,
            "dataType": "DATE",
            "tokenizer": "literal",
            "sortable": true,
            "source": {
                "dateFormats": [
                    "MM/dd/yyyy HH:mm:ss.SSS"
                ],
                "paths": [
                    "metadata.timestamp"
                ],
                "combine": "FIRST"
            },
            "retrieveByDefault": true
        },

Only the date portion of the dateFormat is fixed. The time portion must reflect the format found in the incoming logSource.

The Generic annotator Insight® Pack defines the Annotation Query Language (AQL) that supports log files with no date in the timeOnly splitter. The timeOnly splitter is defined in the file:
GAInsightPack_<version>/extractors/ruleset/timeOnlySplitter