What can be better that a warm place on a cold day?  I've lived in North Carolina in the United States my whole life, and except for the occasional ice storm, it never really gets very cold.  However, I do enjoy a nice fire in the fireplace, if for nothing more that toasting marshmallows when the coals die down.  And as more homes turn to gas logs, finding suppliers of wood logs is getting harder.

 

 

 

Fortunately for IBM Tivoli Monitoring System p agent users, finding where the logs are is pretty simple.  All the logs will be in the <install dir>/logs directory.  But which logs should you look at - there can be lots of them!

 

 

 

First of all, like a many Tivoli Monitoring agents, each System p agent consists of two primary processes:

 

1. kxxagent - what we call the"factory agent," where xx is:
• px - AIX Premium agent
• va - VIOS Premium agent
• pk - CEC Base agent
• ph - HMC Base agent
2. yyyDataProvider[-zz] - this is the data provider,  where yyy is:
   
• aix - for the AIX and VIOS agents, and xx is 53 on AIX 5.3 and 6.1 on AIX 6.1 and above
  
• cec for the CEC agent,
  
• hmc for HMC agent

For example, the VIOS Premium agent will start the kvaagent process, which will start the aixDataProvider-61 process.

 

The kxxagent process is responsible for getting data from the data provider, formatting it and forwarding it to the Tivoli Enterprise Monitoring Server.  The data provider is responsible for collecting the metrics and sending them to kxxagent.  Each process has its own set of logs, but most of the work is done in the data provider, which of course is where most of the problems can occur.

 

The kxxagent process will create logs with these names:

<hostname>_xx_kxxagent_<timestamp>_<YY>.log, where YY is a sequential number starting at 01.  For the AIX Premium agent (px) running on a system with hostname "myhost" with a hexadecimal timestamp of "5122FB32," the first log file name will be:

 

myhost_px_kpxagent_5122FB32_01.log

 

The data provider process will create a log file with a similar name, but instead of kxxagent, the data provider process name will be used.  Again taking the case above, the data provider log file name will be:

myhost_px_aixDataProvider-61_5122FB34_01.log

 

Notice that the timestamp might not match exactly.  This is because it can take a couple of seconds for the kxxagent to start the data provider, so the timestamps might be off by a second or two.

Now that you know where the logs are and what they are named, what can you learn from them?  Usually, you don't care what is in them unless you are having a problem with an agent.  A major concern would be if you started the agent, but you are not seeing data in the Tivoli Enterprise Portal.  There can be lots of reasons for this, and investigating the logs can help.  If the agent is having trouble getting data, then the data provider log should tell you why.  If it can get data, but the kxxagent is not forwarding the data to the Tivoli Enterprise Console (via the Tivoli Enterprise Management Server), then the kxxagent log is the place to look.

 

An example of a problem that is becoming rarer is the inability of the data provider to initialize the System Performance Measurement Interface (SPMI) of the AIX Performance Tools from the perfagent.tools fileset.  If you have the latest fileset installed, this should not be a problem, but it used to happen more frequently (see technote on Required perfagent.tools versions).  In this case, the data provider log would contain the string "ERROR in initializing spmi:" and it would be one of the last messages in the log, since the data provider would stop at that point.

 

To see more details on logging and other helpful information, go to the System p Agents Troubleshooting Wiki.