Topic
8 replies Latest Post - ‏2013-08-13T22:57:14Z by GER_MCC
JRPTexas
JRPTexas
28 Posts
ACCEPTED ANSWER

Pinned topic One more try with the Generic Annotation Pack

‏2013-08-13T14:21:40Z |

I am testing with a different log file now. I am still not able to ingest any log events. I feel like I am skipping a step.  Here is my workflow:

1) I created a new Source Type. The source type uses "Generic-dateTime-Split as the splitter and Generic-Annotate as the annotator. I copied and pasted the index configuration from the Generic source type into my new source type, changing the date format to match my log sample.

2) I created a new Collection. The collection uses the new source type.

3) I created a new log source. The log source uses the new source type and the new collection.

4) I copied my log file into the /home/loguser/IBM/LogAnalyticsWorkgroup/logsources/GAInsightPack/ directory.

After these steps, the UnityEIFReceiver and GenericReceiver logs show that the product is loogin at the collection and the log source, but they don't ingest any data or events.

I think I am missing a step or two in getting a log file to pass data through the GA insight pack.

 

Here are a few lines from my log sample, just to show the time format:

THREAD: OperationRunner-000-NETCOOL,[2013/07/22,20:01:02.671],ERROR,MSG:Socket connect failed, retrying connect
THREAD: OperationRunner-000-NETCOOL,[2013/08/09,14:25:06.329],FINEST,MSG:getDOM() - invoking DocumentBuilder.parse()
THREAD: OperationRunner-001-NETCOOL,[2013/08/09,14:25:06.329],FINEST,MSG:getDOM() - invoking DocumentBuilder.parse()
THREAD: OperationRunner-000-NETCOOL,[2013/08/09,14:25:06.330],FINEST,MSG:getDOM() - DOM document created
THREAD: OperationRunner-001-NETCOOL,[2013/08/09,14:25:06.330],FINEST,MSG:getDOM() - DOM document created
THREAD: OperationRunner-000-NETCOOL,[2013/08/09,14:25:06.417],FINEST,MSG:found: config.version.FIND-BEGIN adding Version
THREAD: OperationRunner-000-NETCOOL,[2013/08/09,14:25:06.417],FINEST,MSG:found: config.version.FIND-END adding ,
THREAD: OperationRunner-000-NETCOOL,[2013/08/09,14:25:06.418],FINEST,MSG:found: model.FIND-BEGIN adding isco

Updated on 2013-08-13T16:23:14Z at 2013-08-13T16:23:14Z by GER_MCC
  • GER_MCC
    GER_MCC
    25 Posts
    ACCEPTED ANSWER

    Re: One more try with the Generic Annotation Pack

    ‏2013-08-13T16:29:02Z  in response to JRPTexas

    I can;t see that the EIF Receiver has posted any records. Have you configured either the Log File Agent or the Data Collector client to read your file.

    LFA is a useful option if the data is to be streamed from the log file. But is this is a once off test, the easiest approach is to use the Data Collector client. Here is the Document reference: https://pic.dhe.ibm.com/infocenter/tivihelp/v3r1/index.jsp?topic=%2Fcom.ibm.iwa.doc_1.0%2Fic-homepage.html

    • JRPTexas
      JRPTexas
      28 Posts
      ACCEPTED ANSWER

      Re: One more try with the Generic Annotation Pack

      ‏2013-08-13T16:47:58Z  in response to GER_MCC

      I haven't changed the configuration of the LFA. I didn't think I would need to (I thought the LFA config came from the GA pack). Should I change the default lfageneric.conf and lfageneric.fmt files?

       

      If I use the REST client to load log file information, how should I create my source type and log source?

      • GER_MCC
        GER_MCC
        25 Posts
        ACCEPTED ANSWER

        Re: One more try with the Generic Annotation Pack

        ‏2013-08-13T22:25:29Z  in response to JRPTexas

        As long as your file is under the logsources/GAInsightPack directory, there should be no change required to IBM-LFA-6.30/config/lo/GAInsightPack-lfageneric.conf.

        There is no change required to the log source and source type for loading from REST client. Using the REST client makes it easier to determine if the data has been injested as the result is displayed in the console.

    • GER_MCC
      GER_MCC
      25 Posts
      ACCEPTED ANSWER

      Re: One more try with the Generic Annotation Pack

      ‏2013-08-13T18:16:20Z  in response to GER_MCC

      To see how much data has been loaded for each Log Source, use the following command

       <Install dir>/IBM/LogAnalyticsWorkgroup/utilities/export_statistics -u unityadmin -p unityadmin https://localhost:9987/Unity/

      replacing localhost with an actual hostname or IP address as required.

      The output shows the number of bytes injested, as below:

      Log Source     |      Collection      |     Date     |  Ingested Bytes  |  Deleted Bytes  |  Total Bytes  |           Log Path           |  Hostname  |  Deleted
      DT_DB2_db2diag |  DT_DB2_db2diag_C    |  2013-08-02  |       5069       |        0        |      5069     |  DayTrader/dt_db2diag.txt    |  cldegd60  |   false


      If the Data Collector client is used to input a log file, the data is sent directly to the GenericReceiver. If LFA is used, then the data goes from LFA, to EIF Receiver to Generic Receiver.
       

  • dmcclure
    dmcclure
    98 Posts
    ACCEPTED ANSWER

    Re: One more try with the Generic Annotation Pack

    ‏2013-08-13T16:37:39Z  in response to JRPTexas

    Those are all the steps I've done 100's of times with positive working results.

    Has anything worked with your installation? Is this v1.1.0.1 or v1.1?

     

    Doug

    • JRPTexas
      JRPTexas
      28 Posts
      ACCEPTED ANSWER

      Re: One more try with the Generic Annotation Pack

      ‏2013-08-13T16:54:18Z  in response to dmcclure

      This is v1.1.0.1. The pack versions are:

      GAInsightPack = 1.1.0

      DB2InsightPack = 1.1.0

      WASInsightPack = 1.1.0

       

      I have got DB2 and WAS files to ingest with no problem using their respective packs. I haven't had any success loading logs using the Generic Annotation pack.

      I am going back and watching the GO training to see if I am missing a step to using the Generic Annotation pack.

      • GER_MCC
        GER_MCC
        25 Posts
        ACCEPTED ANSWER

        Re: One more try with the Generic Annotation Pack

        ‏2013-08-13T19:51:31Z  in response to JRPTexas

        I think the issue is still with the date formats. When I use formats as per the documentation, for example, [25/07/13 20:01:02.671], I can analyse the file ok in eclipse with tooling. But I am not able to load with date format starting with a year and comma seperating date and time.

        I have not tried with LFA. I am getting an error at present when I try load with Data Collector. The generic receiver is reporting an error:

        "errorMessage":"CTGLA5121E : Invalid index data for collection \"testCollection\""}],"batchSize":3,"writeTime":"2013-08-13T15:39:40.113-0400","numFailures":3},"RESPONSE_MESSAGE":"INPUT_BATCH_PROCESSED","RESPONSE_CODE":200}

        I am investigating this.

        • GER_MCC
          GER_MCC
          25 Posts
          ACCEPTED ANSWER

          Re: One more try with the Generic Annotation Pack

          ‏2013-08-13T22:57:14Z  in response to GER_MCC

          My issue above was also caused by time format issues. When I reverted to standard time format, the file worked fine. My colleague will have an update for you on Forum Thread "Question about GA Insight Pack" with AQL changes which will allow for a wider range of date formats.