IBM Support

ITM Agent Insights: Monitoring logs with ITM: Initial problem determination steps for Log File Agent - LO

Technical Blog Post


Abstract

ITM Agent Insights: Monitoring logs with ITM: Initial problem determination steps for Log File Agent - LO

Body

This post is intended to provide a beginning set of steps for gathering initial documentation to begin working a ticket when engaging IBM Support for problems being reported with Log File Agent - LO.


The external symptom typically reported is the LO agent is not populating data in the TEP portal as expected.


#1) Provide details to confirm overall ITM environment.
The first step is confirming the ITM environment as different levels of LO agent may allow for different configuration parameters.


The LO agent running on the endpoint should be the same level as the LO agent application support on ITM infrastructure systems - TEMS / TEPS / TEP Desktop.

 

Gather ITM environmental information from the environment:
1) OS platform, hostname, and ITM components installed on the TEPS system.
If this is Windows, provide "kincinfo -d", if Unix/Linux provide "cinfo -i" to show the ITM components and application support levels

 

2) OS platform, hostname, and ITM components installed on the HUB TEMS (and Remote TEMS if using them).
Again, if they are running on Windows OS, provide "kincinfo -d", if on Unix/Linux provide "cinfo -i"

3) OS platform, hostname, and ITM components installed on TEP Desktop system (if using TEP Desktop installed separately from TEPS)
Again, if the agent system is on Windows OS, provide "kincinfo -d", if they are on UNIX/Linux provide "cinfo -i"

4) OS platform, hostname, and ITM components installed on the agent endpoint.

Provide output from running "pdcollect" utility on the agent system. 

Run "pdcollect" with appropriate authority so that "cinfo.info" file included in the "pdcollect" output file is populated with ITM installation details.

If the "cinfo.info" indicates "insufficient privileges" you should re-run the utility to make sure the output file contains the necessary details.

On UNIX / Linux this is usually accomplished by running "pdcollect" as "root".

On Windows, this requires using "Run As Administrator" even using a local administrator account.

 

 

#2) Confirm the configuration for the LO agent instance.
The LO agent allows for various configurations - "base" instance, subnodes utilizing autodiscovery, or a combination of both "base" and "subnode" configuration.

The LO agent relies on user generated CONF (.conf) and FORMAT (.fmt) files.

 

Review configuration information from the environment through the MTEMS gui, or from "pdcollect" to get copies of the .cfg file.
*NOTE* The "pdcollect" utility does not gather the CONF / FORMAT file for the "base" instance, and does not gather the .conf / .fmt file pairs from the autodiscovery directory.  These files need to be provided manually.

 

Example walk-through configuring an LO agent instance through the MTEMS gui:.
Bring up the MTEMS gui, and right-click on the "template" entry to configure a new instance:

image

Give the instance a name:

image

Provide the details for the "Log File Adapter Configuration" panel contents.  This is for the "base" LO agent instance.

imageIf specifying Conf / Format file values, provide copies of the files that match the specification.

 

In the "Log File Adapter Global Settings" configuration panel, confirm the value of the autodiscovery directory specified:

image

 

 

Example walk-through configuring an LO agent instance from command line on UNIX / Linux:

# cd <candle home>/bin
# ./itmcmd config -A lo
Agent configuration started...
Enter instance name (default is: ): 51494LO   <--- LO agent instance name
Edit "Tivoli Log File Agent" settings? [ 1=Yes, 2=No ] (default is: 1): 1
Edit 'Log File Adapter Configuration' settings? [ 1=Yes, 2=No ] (default is: 1): 1
Conf file (default is: ):               <--- Conf file for the "base" instance - KLO_CONF_FILE.  This value can be left blank if relying on subnodes and autodiscovery.
Format File (default is: ):           <--- Format file for the "base" instance - KLO_FORMAT_FILE.  This value can be left blank if relying on subnodes and autodiscovery.
Send EIF Events to OMNIbus [ 1=Yes, 2=No ] (default is: 1): 2
Send ITM Events [ 1=Yes, 2=No ] (default is: 1): 1
Automatically initialize UNIX syslog [ 1=Yes, 2=No, 3=Use .conf file value ] (default is: 3): 3
Edit 'Log File Adapter Global Settings' settings? [ 1=Yes, 2=No ] (default is: 1): 1
Process Priority Class [ 1=A, 2=B, 3=C, 4=D, 5=E, 6=F, 7=Use .conf file value ] (default is: 7): 7
Process maximum CPU percentage (default is: 100):
Configuration file autodiscovery directory (default is: ${CANDLE_HOME}/config/lo):         <-- The autodiscovery directory to search for .conf / .fmt pairs - KLO_FILE_DISCOVERY_DIR.

Will this agent connect to a TEMS? [1=YES, 2=NO] (Default is: 1): 1
TEMS Host Name (Default is: NMP136): test.system.ibm.com

Network Protocol [ip, sna, ip.pipe or ip.spipe] (Default is: ip.pipe):

     Now choose the next protocol from one of these:
     - ip
     - sna
     - ip.spipe
     - 0 for none
Network Protocol 2 (Default is: 0):
IP.PIPE Port Number (Default is: 1918):
Enter name of KDC_PARTITION (Default is: null):

Configure connection for a secondary TEMS? [1=YES, 2=NO] (Default is: 2):
Enter Optional Primary Network Name or 0 for "none" (Default is: 0):
Agent configuration completed...

 

If there are any .conf / .fmt files under the autodiscovery directory or one of its subdirectories, manually gather and provide copies of these files.
When configuring the LO agent to use autodiscovery, provide the output of "dir /S" on Windows or "ls -laR" on UNIX/Linux to show the contents of the autodiscovery directory/subdirs.

 

When reviewing configuration details from "pdcollect", in the LO agent instance .cfg file, confirm the values specified for:
KLO_FORMAT_FILE=
KLO_CONF_FILE=
KLO_FILE_DISCOVERY_DIR=

 

Manually provide copies of any file specified by KLO_FORMAT_FILE= and KLO_CONF_FILE=, and provide all .conf / .fmt files under the value specified as KLO_FILE_DISCOVERY_DIR=.


*NOTE* It is important to get actual copies of the files to provide to Support, as it may be necessary to review the files in a hexadecimal text editor to make sure that there are not any non-ASCII characters in the config or format file that might be causing an issue with monitoring a log.

 

 

#3) Review the "Data Collection Status" workspace view in the TEP:

Confirm if the desired log to monitor is found and has Object Status  "ACTIVE" and File Status "OK".

imageProvide screen capture from the TEP showing the full TEP window for the "Data Collection Status" workspace.

 

#4) Confirm that log entries are being parsed from the monitored log.

If logs are being monitored, and the status is OK, but log entries as not showing up as expected, confirm if log entries are being written to the unmatch log, and whether the values in the "Monitored File Status" workspace are being updated for the log.

The values for  Number of Records Matched / Number of Records Not Matched / Number of Records Processed / Current File Position should be changing if the LO agent is detecting newly written data  and is processing those log entries against the stanzas in the FORMAT (.fmt) file.

Below is an example that simulates new entries being written while the LO agent is monitoring "nas_quotas.txt" file.

Initially there is no "new" data to process since the LO agent has been started as the number of records processed / matched / not matched are all zero.

image

Refreshing the "Data Collection Status" workspace after new data has been written to the monitored log shows the values updated to reflect the number of records processed and how many matched / didn't match:

image

Review the unmatch log to confirm the entries that are not matching against any RegEx stanza in the .fmt file.  Any entries that are expected to match should be reviewed against the regular expressions to determine why the are not matching as expected.  The Num Records Not Matched should be equal to the values written to the unmatch log assuming the unmatch log was cleared since the last time log monitoring was restarted for monitored log.

image

 

The "Logfile Events (v6)" workspace is populated with the matching entries:

image

A clear description of the problem being reported is necessary to understand if a log is not being monitored, or if the log appears to be monitored but entries are not matching as expected, or where the log entry is displayed but not parsed as desired into the correct attribute values.

For debugging, it is ALWAYS recommended that an unmatch log be specified using UnmatchLog parameter in the CONF (.conf) file.

 

*NOTE* The default workspace query limits results to 100 rows of data.  If a "missing" log entry is being counted in the "Num Records Matched" value, and is not showing up in the unmatch log, make sure that the reason it is not displayed in the TEP is due to the workspace only displaying the first 100 rows of data.

image

For debugging, it is recommended to edit the query to modify the query that populates the workspace view to return all rows of data.

 

*NOTE* If the FORMAT (.fmt) contains any stanzas using *DISCARD* stanzas, the log entries that match against these stanzas are "discarded" log entries.  Discarded log entries are not displayed in the Logfile Events workspace AND they do not show up in the unmatch log since they did match a stanza, albeit a *DISCARD* stanza.

 

If reporting a problem with LO agent, provide environment / configuration details as well as a clear verbal description if the issue is log entries are not being seen in the TEP.

Providing screen captures of the workspaces as in above examples to assist L2 Support understand the nature of the issue.

 

Submitter: drd401709
Compid: 5724c04lf
Reference DCF technotes:

1679044 - Initial problem determination steps for Log File Agent - LO.

Keywords:
LFA LO kloagent

 

Additional ITM Agent Insights series of IBM Tivoli Monitoring Agent blogs are indexed under ITM Agent Insights: Introduction.

 

Tutorials Point

Subscribe and follow us for all the latest information directly on your social feeds:

 

imageimageimage

Check out all our other posts and updates:

Academy Blogs
Academy Videos
Academy Google+
Academy Twitter

image

 

[{"Business Unit":{"code":"BU004","label":"Hybrid Cloud"},"Product":{"code":"","label":""},"Component":"","Platform":[{"code":"","label":""}],"Version":"","Edition":""}]

UID

ibm11083021