Topic
  • 8 replies
  • Latest Post - ‏2013-08-13T22:57:14Z by GER_MCC
JRPTexas
JRPTexas
28 Posts

Pinned topic One more try with the Generic Annotation Pack

‏2013-08-13T14:21:40Z |

I am testing with a different log file now. I am still not able to ingest any log events. I feel like I am skipping a step.  Here is my workflow:

1) I created a new Source Type. The source type uses "Generic-dateTime-Split as the splitter and Generic-Annotate as the annotator. I copied and pasted the index configuration from the Generic source type into my new source type, changing the date format to match my log sample.

2) I created a new Collection. The collection uses the new source type.

3) I created a new log source. The log source uses the new source type and the new collection.

4) I copied my log file into the /home/loguser/IBM/LogAnalyticsWorkgroup/logsources/GAInsightPack/ directory.

After these steps, the UnityEIFReceiver and GenericReceiver logs show that the product is loogin at the collection and the log source, but they don't ingest any data or events.

I think I am missing a step or two in getting a log file to pass data through the GA insight pack.

 

Here are a few lines from my log sample, just to show the time format:

THREAD: OperationRunner-000-NETCOOL,[2013/07/22,20:01:02.671],ERROR,MSG:Socket connect failed, retrying connect
THREAD: OperationRunner-000-NETCOOL,[2013/08/09,14:25:06.329],FINEST,MSG:getDOM() - invoking DocumentBuilder.parse()
THREAD: OperationRunner-001-NETCOOL,[2013/08/09,14:25:06.329],FINEST,MSG:getDOM() - invoking DocumentBuilder.parse()
THREAD: OperationRunner-000-NETCOOL,[2013/08/09,14:25:06.330],FINEST,MSG:getDOM() - DOM document created
THREAD: OperationRunner-001-NETCOOL,[2013/08/09,14:25:06.330],FINEST,MSG:getDOM() - DOM document created
THREAD: OperationRunner-000-NETCOOL,[2013/08/09,14:25:06.417],FINEST,MSG:found: config.version.FIND-BEGIN adding Version
THREAD: OperationRunner-000-NETCOOL,[2013/08/09,14:25:06.417],FINEST,MSG:found: config.version.FIND-END adding ,
THREAD: OperationRunner-000-NETCOOL,[2013/08/09,14:25:06.418],FINEST,MSG:found: model.FIND-BEGIN adding isco

Updated on 2013-08-13T16:23:14Z at 2013-08-13T16:23:14Z by GER_MCC
  • GER_MCC
    GER_MCC
    25 Posts
    ACCEPTED ANSWER

    Re: One more try with the Generic Annotation Pack

    ‏2013-08-13T22:57:14Z  
    • GER_MCC
    • ‏2013-08-13T19:51:31Z

    I think the issue is still with the date formats. When I use formats as per the documentation, for example, [25/07/13 20:01:02.671], I can analyse the file ok in eclipse with tooling. But I am not able to load with date format starting with a year and comma seperating date and time.

    I have not tried with LFA. I am getting an error at present when I try load with Data Collector. The generic receiver is reporting an error:

    "errorMessage":"CTGLA5121E : Invalid index data for collection \"testCollection\""}],"batchSize":3,"writeTime":"2013-08-13T15:39:40.113-0400","numFailures":3},"RESPONSE_MESSAGE":"INPUT_BATCH_PROCESSED","RESPONSE_CODE":200}

    I am investigating this.

    My issue above was also caused by time format issues. When I reverted to standard time format, the file worked fine. My colleague will have an update for you on Forum Thread "Question about GA Insight Pack" with AQL changes which will allow for a wider range of date formats.

  • GER_MCC
    GER_MCC
    25 Posts

    Re: One more try with the Generic Annotation Pack

    ‏2013-08-13T16:29:02Z  

    I can;t see that the EIF Receiver has posted any records. Have you configured either the Log File Agent or the Data Collector client to read your file.

    LFA is a useful option if the data is to be streamed from the log file. But is this is a once off test, the easiest approach is to use the Data Collector client. Here is the Document reference: https://pic.dhe.ibm.com/infocenter/tivihelp/v3r1/index.jsp?topic=%2Fcom.ibm.iwa.doc_1.0%2Fic-homepage.html

  • dmcclure
    dmcclure
    103 Posts

    Re: One more try with the Generic Annotation Pack

    ‏2013-08-13T16:37:39Z  

    Those are all the steps I've done 100's of times with positive working results.

    Has anything worked with your installation? Is this v1.1.0.1 or v1.1?

     

    Doug

  • JRPTexas
    JRPTexas
    28 Posts

    Re: One more try with the Generic Annotation Pack

    ‏2013-08-13T16:47:58Z  
    • GER_MCC
    • ‏2013-08-13T16:29:02Z

    I can;t see that the EIF Receiver has posted any records. Have you configured either the Log File Agent or the Data Collector client to read your file.

    LFA is a useful option if the data is to be streamed from the log file. But is this is a once off test, the easiest approach is to use the Data Collector client. Here is the Document reference: https://pic.dhe.ibm.com/infocenter/tivihelp/v3r1/index.jsp?topic=%2Fcom.ibm.iwa.doc_1.0%2Fic-homepage.html

    I haven't changed the configuration of the LFA. I didn't think I would need to (I thought the LFA config came from the GA pack). Should I change the default lfageneric.conf and lfageneric.fmt files?

     

    If I use the REST client to load log file information, how should I create my source type and log source?

  • JRPTexas
    JRPTexas
    28 Posts

    Re: One more try with the Generic Annotation Pack

    ‏2013-08-13T16:54:18Z  
    • dmcclure
    • ‏2013-08-13T16:37:39Z

    Those are all the steps I've done 100's of times with positive working results.

    Has anything worked with your installation? Is this v1.1.0.1 or v1.1?

     

    Doug

    This is v1.1.0.1. The pack versions are:

    GAInsightPack = 1.1.0

    DB2InsightPack = 1.1.0

    WASInsightPack = 1.1.0

     

    I have got DB2 and WAS files to ingest with no problem using their respective packs. I haven't had any success loading logs using the Generic Annotation pack.

    I am going back and watching the GO training to see if I am missing a step to using the Generic Annotation pack.

  • GER_MCC
    GER_MCC
    25 Posts

    Re: One more try with the Generic Annotation Pack

    ‏2013-08-13T18:16:20Z  
    • GER_MCC
    • ‏2013-08-13T16:29:02Z

    I can;t see that the EIF Receiver has posted any records. Have you configured either the Log File Agent or the Data Collector client to read your file.

    LFA is a useful option if the data is to be streamed from the log file. But is this is a once off test, the easiest approach is to use the Data Collector client. Here is the Document reference: https://pic.dhe.ibm.com/infocenter/tivihelp/v3r1/index.jsp?topic=%2Fcom.ibm.iwa.doc_1.0%2Fic-homepage.html

    To see how much data has been loaded for each Log Source, use the following command

     <Install dir>/IBM/LogAnalyticsWorkgroup/utilities/export_statistics -u unityadmin -p unityadmin https://localhost:9987/Unity/

    replacing localhost with an actual hostname or IP address as required.

    The output shows the number of bytes injested, as below:

    Log Source     |      Collection      |     Date     |  Ingested Bytes  |  Deleted Bytes  |  Total Bytes  |           Log Path           |  Hostname  |  Deleted
    DT_DB2_db2diag |  DT_DB2_db2diag_C    |  2013-08-02  |       5069       |        0        |      5069     |  DayTrader/dt_db2diag.txt    |  cldegd60  |   false


    If the Data Collector client is used to input a log file, the data is sent directly to the GenericReceiver. If LFA is used, then the data goes from LFA, to EIF Receiver to Generic Receiver.
     

  • GER_MCC
    GER_MCC
    25 Posts

    Re: One more try with the Generic Annotation Pack

    ‏2013-08-13T19:51:31Z  
    • JRPTexas
    • ‏2013-08-13T16:54:18Z

    This is v1.1.0.1. The pack versions are:

    GAInsightPack = 1.1.0

    DB2InsightPack = 1.1.0

    WASInsightPack = 1.1.0

     

    I have got DB2 and WAS files to ingest with no problem using their respective packs. I haven't had any success loading logs using the Generic Annotation pack.

    I am going back and watching the GO training to see if I am missing a step to using the Generic Annotation pack.

    I think the issue is still with the date formats. When I use formats as per the documentation, for example, [25/07/13 20:01:02.671], I can analyse the file ok in eclipse with tooling. But I am not able to load with date format starting with a year and comma seperating date and time.

    I have not tried with LFA. I am getting an error at present when I try load with Data Collector. The generic receiver is reporting an error:

    "errorMessage":"CTGLA5121E : Invalid index data for collection \"testCollection\""}],"batchSize":3,"writeTime":"2013-08-13T15:39:40.113-0400","numFailures":3},"RESPONSE_MESSAGE":"INPUT_BATCH_PROCESSED","RESPONSE_CODE":200}

    I am investigating this.

  • GER_MCC
    GER_MCC
    25 Posts

    Re: One more try with the Generic Annotation Pack

    ‏2013-08-13T22:25:29Z  
    • JRPTexas
    • ‏2013-08-13T16:47:58Z

    I haven't changed the configuration of the LFA. I didn't think I would need to (I thought the LFA config came from the GA pack). Should I change the default lfageneric.conf and lfageneric.fmt files?

     

    If I use the REST client to load log file information, how should I create my source type and log source?

    As long as your file is under the logsources/GAInsightPack directory, there should be no change required to IBM-LFA-6.30/config/lo/GAInsightPack-lfageneric.conf.

    There is no change required to the log source and source type for loading from REST client. Using the REST client makes it easier to determine if the data has been injested as the result is displayed in the console.

  • GER_MCC
    GER_MCC
    25 Posts

    Re: One more try with the Generic Annotation Pack

    ‏2013-08-13T22:57:14Z  
    • GER_MCC
    • ‏2013-08-13T19:51:31Z

    I think the issue is still with the date formats. When I use formats as per the documentation, for example, [25/07/13 20:01:02.671], I can analyse the file ok in eclipse with tooling. But I am not able to load with date format starting with a year and comma seperating date and time.

    I have not tried with LFA. I am getting an error at present when I try load with Data Collector. The generic receiver is reporting an error:

    "errorMessage":"CTGLA5121E : Invalid index data for collection \"testCollection\""}],"batchSize":3,"writeTime":"2013-08-13T15:39:40.113-0400","numFailures":3},"RESPONSE_MESSAGE":"INPUT_BATCH_PROCESSED","RESPONSE_CODE":200}

    I am investigating this.

    My issue above was also caused by time format issues. When I reverted to standard time format, the file worked fine. My colleague will have an update for you on Forum Thread "Question about GA Insight Pack" with AQL changes which will allow for a wider range of date formats.