Loading a batch of log files with the Log File Agent

Use this example to help you to understand how to use the Log File Agent to load log a batch of files.

To use the log file agent to load log information, complete the following steps:

1. In the Administrative Settings page, define an appropriate log file source.
2. Ensure that the log file you want to add is in the appropriate directory.
   For WAS logs, place the log file in the following directory:

   <HOME>/IBM/LogAnalysis/logsources/WASInsightPack

   For DB2 logs, place the log file in the following directory:

   <HOME>/IBM/LogAnalysis/logsources/DB2InsightPack

   For Generic annotator log files, place the log file in the following directory:

   $UNITY_HOME/logsources/GAInsightPack

   The log file is automatically picked up and analyzed. Depending on the size of the log file, processing it could take some time.
3. Optional: To monitor progress, check the following log files:
   • <HOME>/IBM/LogAnalysis/logs/GenericReceiver.log
   • <HOME>/IBM/LogAnalysis/logs/UnityEifReceiver.log

   When you are using the Log File Agent to perform data collection, monitor the UnityEIFReceiver.log and GenericReceiver.log log files located in the $UNITY_HOME/logs directory to ensure that the data ingestion has completed correctly.

   This example illustrates the addition of a batch of log records. The result is indicated in the RESPONSE MESSAGE section of the log file:

   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   2013-04-20 04:43:10,032 [pool-5-thread-1] INFO  - LogEventPoster : -
   Posting Event to UNITY DATA COLLECTOR -
      https://nc9118041070:9987/Unity/DataCollector
   2013-04-20 04:43:24,273 [pool-5-thread-1] INFO  - LogEventPoster :
      +++++++++ RESPONSE MESSAGE +++++++++
   2013-04-20 04:43:24,273 [pool-5-thread-1] INFO  - LogEventPoster : OK
   2013-04-20 04:43:24,273 [pool-5-thread-1] INFO  - LogEventPoster :
      {    "batchSize": 2078,    
   "failures": [    ],    "numFailures": 0 }
   2013-04-20 04:43:24,273 [pool-5-thread-1] INFO  - LogEventPoster :
       ++++++++++++++++++++++++++++++++++++
   2013-04-2 04:43:24,273 [pool-5-thread-1] INFO  - LogEventPoster :
       EIF event delivery to Generic Receiver -- SUCCESS
   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

   In this log, the number of log records processed is indicated in the line:

   {    "batchSize": 2078,    "failures": [    ],    "numFailures": 0 }

   2078 log records were successfully ingested. The numFailures value indicates the number of failures in the ingestion of the log records. For example, a value of 5 for the numFailures value indicates that 5 log records were not ingested.

   When data collection has completed, if the EIF Receiver buffer is partially filled, any remaining log records are posted to the Generic Receiver. This is recorded in the log as a TIMEOUT FLUSH event. These events are added to the log file at the end of the session of data collection:

   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   
   2013-04-20 04:54:26,341 [pool-4-thread-1] INFO  - LogEventService :
    TIMEOUT FLUSH for logsource:nc9118041070::
     /home/example/LogAnalytics/logsources/
   WASInsightPack/TipTrace5.log
   2013-04-20 04:54:26,359 [pool-5-thread-1] INFO  - LogEventPoster : ---
   Posting Event to UNITY DATA COLLECTOR -
      https://nc9118041070:9987/Unity/DataCollector
   2013-04-20 04:54:38,581 [pool-5-thread-1] INFO  - LogEventPoster :
      +++++++++ RESPONSE MESSAGE +++++++++
   2013-04-20 04:54:38,582 [pool-5-thread-1] INFO  - LogEventPoster : OK
   2013-04-20 04:54:38,582 [pool-5-thread-1] INFO  - LogEventPoster :
      {    "batchSize": 1714,   
   "failures": [    ],    "numFailures": 0 }
   2013-04-20 04:54:38,582 [pool-5-thread-1] INFO  - LogEventPoster :
      ++++++++++++++++++++++++++++++++++++
   2013-04-20 04:54:38,582 [pool-5-thread-1] INFO  - LogEventPoster :
      EIF event delivery to Generic Receiver -- SUCCESS
   2013-04-20 04:54:38,583 [pool-4-thread-1] INFO  - LogEventService :
      POST RESULT:
   {"failures":[],"batchSize":1714,"numFailures":0}
   
   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

   To calculate the number of events that have been processed, calculate the sum of all of the batchSize values. To calculate the number of events ingested, calculate the sum of all of the batchSize values and deduct the total sum of numFailure values.

   If the ingestion fails, an error message is recorded in the UnityEIFReceiver.log:

   2013-05-16 02:16:11,184 [pool-7-thread-1] INFO  - LogEventPoster :
      +++++++++ RESPONSE MESSAGE +++++++++
   2013-05-16 02:16:11,184 [pool-7-thread-1] INFO  - LogEventPoster : Not Found
   2013-05-16 02:16:11,184 [pool-7-thread-1] INFO  - LogEventPoster :
      {"BATCH_STATUS":"NONE","RESPONSE_MESSAGE":
   "CTGLA0401E : Missing log source ","RESPONSE_CODE":404}
   2013-05-16 02:16:11,184 [pool-7-thread-1] INFO  - LogEventPoster :
      ++++++++++++++++++++++++++++++++++++
   2013-05-16 02:16:11,184 [pool-7-thread-1] INFO  - LogEventPoster :
      FAILURE -  ResponseCode:404 ResponseMessage:Not Found

   Additional HTTP response codes are as follows:

   413

   Request Entity Too Large: Displayed if a batch size is greater than the Generic Receiver default value set in the $UNITY_HOME/wlp/usr/servers/Unity/apps/Unity.war/WEB-INF/unitysetup.properties.

   500

   Internal Server Error: Displayed when there is any issue withIBM Operations Analytics Log Analysis such as a database error or any other runtime error.

   404

   Not Found: Displayed when a Log Source is not found for a hostname and log path combination in the request.

   409

   Conflict: Displayed if the data batch is posted for a Log Source that is an inactive state or if there is a conflict between the data posted and the data expected by the server. For example, the inputType field in the request JSON does not match the inputType field in the Collection for the requested hostname and log path combination.

   200

   OK: Displayed when the request is processed by the server. The status of the processed batch of records is returned with the total number of records ingested, how many failed records are present and which failed.

   400®

   Bad Request: Displayed when the request JSON does not contain the required fields expected by the Generic Receiver or where the JSON is not properly formed.