Topic
  • 3 replies
  • Latest Post - ‏2014-01-13T18:58:01Z by MorganDennis
YEKT_Zdenka_Spoljaric
14 Posts

Pinned topic Creating metadata.json for Machine Data

‏2013-10-07T01:53:17Z |

Hi All

 

I would like to create a metadata.json for the TIVOLI MONITORING log files as per the sample below.

I have been using the Machine Data Adapter but so far was not successful.

Could you please assist .

SAMPLE:

1       1377271692      1       "2013-08-23-15.28.12.762244"    "2013-08-23-15.40.19.762244"    "TMW_LowLogicalDiskSpace"       "TMNT"  "IBM Intel Support"     "177.132.8.238" "TEL"   "wszapp1452" "wszapp1452.in.test.com.zz"    30      "gszz_clearing: CLOSED event."  30      727     "The logical drive C: has very little free space (         5 percent)." 0       0       1       0       0   20130823                                                 "TEL"   "W32"   "IBM Intel Support"
1       1377271919      1       "2013-08-23-15.32.00.059076"    "2013-08-23-15.34.30.059076"    "InterfaceNotOperat"    "TMNT"  "PRE_PRODUCTION"        "133.144.36.59" "TEL"   "lxzapp0247"     "lxap
p0247.edw.in.test.com.zz"    30      "gszz_clearing: CLOSED event."  30      150     "The interface eth0 is enabled, but not operational."   0       0       1       0       0       20130823    "TEL"    "LNX"   "PRE_PRODUCTION"
1       1376839161      1       "2013-08-18-15.19.21.149453"    "2013-08-18-16.04.42.149453"    "TMW_MemoryLeakInPB"    "TMNT"  "PRE_PRODUCTION"        "193.74.154.254"        "TEL"   "myser0068c01"
        "myser0068c01.in.test.com.zz" 30      "gszz_clearing: CLOSED event."  30      2721    "The private memory of the process opcmsga (process ID       1612) has a memory leak."  0       0   10       0       20130818                                                        "TEL"   "W32"   "PRE_PRODUCTION"
 

  • ClausSamuelsen
    ClausSamuelsen
    4 Posts

    Re: Creating metadata.json for Machine Data

    ‏2013-10-18T14:38:00Z  

    Have you tried to use the MDA Configuration Tool ?

    You can start it from http://biginsights:8080/mdaconfig/jsp/MDAConfig.jsp

    There are two important fields in the metadata.json file you always must define: "dateTimeFormat", "preTimestampRegex" and "logType".

    dateTimeFormat follows the java SimpeDateFormat definitions (http://docs.oracle.com/javase/1.4.2/docs/api/java/text/SimpleDateFormat.html),

    so in your example it should be dateTimeFormat:"yyyy-MM-dd-HH.mm.ss.SSSSS".

    The preTimestampRegex is important because it describes what is found before the primary timestamp. In your example there are at least two timestamps per log record, so if the first one is the primary timestamp, the regular expression will be something like:

    preTimestampRegex:"(\\n|\\r)\d{1}\s+\d}\s+\d{1}\s+\"" 

    Explanation for the regular expression

    (\\n|\\r) : newline or carriage-return (handles both unix and dos files) followed by

    \d{1} : exactly one digit followed by

    \s+ : one or more white space followed by

    \d{10{10} : exactly 10 digits etc

    \" : an double quote character as the last character before the timestamp.

    Since Tivoli Monitoring log file is not (yet) one of the predefined log, you should use the 'generic' type, so

    logType: "generic"

    The whole metadata.json record should look something like:

    {dateTimeFormat:"yyyy-MM-dd-HH.mm.ss.SSSSS", preTimestampRegex:"(\\n|\\r)\d{1}\s+\d{10}\s+\d{1}\s+\"", logType: "generic", batchId:"TivoliMonitoring"}

     

    I recommend to read these MDA articles: http://www.ibm.com/developerworks/data/library/?sort_by=&show_abstract=true&show_all=&search_flag=&contentarea_by=Information+Management&search_by=IBM+Accelerator+for+Machine+Data+Analytics&product_by=-1&topic_by=-1&industry_by=-1&type_by=All+Types&ibm-search=Search

    Part 1 has a good description of the metadata.json file

  • YEKT_Zdenka_Spoljaric
    14 Posts

    Re: Creating metadata.json for Machine Data

    ‏2013-10-21T00:22:53Z  

    Have you tried to use the MDA Configuration Tool ?

    You can start it from http://biginsights:8080/mdaconfig/jsp/MDAConfig.jsp

    There are two important fields in the metadata.json file you always must define: "dateTimeFormat", "preTimestampRegex" and "logType".

    dateTimeFormat follows the java SimpeDateFormat definitions (http://docs.oracle.com/javase/1.4.2/docs/api/java/text/SimpleDateFormat.html),

    so in your example it should be dateTimeFormat:"yyyy-MM-dd-HH.mm.ss.SSSSS".

    The preTimestampRegex is important because it describes what is found before the primary timestamp. In your example there are at least two timestamps per log record, so if the first one is the primary timestamp, the regular expression will be something like:

    preTimestampRegex:"(\\n|\\r)\d{1}\s+\d}\s+\d{1}\s+\"" 

    Explanation for the regular expression

    (\\n|\\r) : newline or carriage-return (handles both unix and dos files) followed by

    \d{1} : exactly one digit followed by

    \s+ : one or more white space followed by

    \d{10{10} : exactly 10 digits etc

    \" : an double quote character as the last character before the timestamp.

    Since Tivoli Monitoring log file is not (yet) one of the predefined log, you should use the 'generic' type, so

    logType: "generic"

    The whole metadata.json record should look something like:

    {dateTimeFormat:"yyyy-MM-dd-HH.mm.ss.SSSSS", preTimestampRegex:"(\\n|\\r)\d{1}\s+\d{10}\s+\d{1}\s+\"", logType: "generic", batchId:"TivoliMonitoring"}

     

    I recommend to read these MDA articles: http://www.ibm.com/developerworks/data/library/?sort_by=&show_abstract=true&show_all=&search_flag=&contentarea_by=Information+Management&search_by=IBM+Accelerator+for+Machine+Data+Analytics&product_by=-1&topic_by=-1&industry_by=-1&type_by=All+Types&ibm-search=Search

    Part 1 has a good description of the metadata.json file

    Thank you very much for your reply.

    I have been using MDA tools for this type of files, but without success even when applying the input you provided in the response above. The original separator in the file is "tab" , so that might be an issue as well...

    I have  tried lot of combinations ... did not work..

    I had a problem with the oracle.log as well... even though I followed the documented instructions ...

    It seams to me that the MDA tool and the machine data accelerator works very well with webserver logs and syslogs... with the delimited files I have mixed success; depends on the delimiter type and the  timestamp format, the results are not consistent.

    It seams to me the one has to go to the  Eclipse Tools  and develop appropriate and very specific aql for some  types because using generic will not produce the desired outcome.

    Again, thank you for your input.  I really do appreciate. 

  • MorganDennis
    MorganDennis
    7 Posts

    Re: Creating metadata.json for Machine Data

    ‏2014-01-13T18:58:01Z  

    Thank you very much for your reply.

    I have been using MDA tools for this type of files, but without success even when applying the input you provided in the response above. The original separator in the file is "tab" , so that might be an issue as well...

    I have  tried lot of combinations ... did not work..

    I had a problem with the oracle.log as well... even though I followed the documented instructions ...

    It seams to me that the MDA tool and the machine data accelerator works very well with webserver logs and syslogs... with the delimited files I have mixed success; depends on the delimiter type and the  timestamp format, the results are not consistent.

    It seams to me the one has to go to the  Eclipse Tools  and develop appropriate and very specific aql for some  types because using generic will not produce the desired outcome.

    Again, thank you for your input.  I really do appreciate. 

    Is this problem resolved? If not, can you pass along the data set and the issues that you are having with that data set?