ITM Agent Insights: Off-line monitoring agents due to missing application support

Body

Tivoli Monitoring agent is off-line even though the agent process is running on the endpoint, and startup log shows it successfully connected to a TEMS.

Symptom:
The managed system for the monitoring agent is greyed out or completely missing from the navigator tree in the Tivoli Enterprise Portal.

Situation alerts generated for managed system off-line situation - "MS_Offline".
  

Cause:
Missing application support for the type of monitoring agent on the TEMS the endpoint is connecting to.
   

Environment:
Generic issue for Tivoli Monitoring for all agents and on all platforms.
   

Diagnosis:
Confirm that the monitoring agent process is running:
Check "Task Manager" on Windows endpoints, or "ps -ef" on UNIX / Linux endpoints.
 
Review the monitoring agent's .LG0 file to confirm it shows a connection to TEMS:
   KRAREG000 Connecting to CMS <TEMS_name>

Also confirm whether there is indication that Self-Describing Agent function is disabled:

   KRAA0013  Self-Describing Agent function disabled by TEMS "<TEMS_name>" for PRODUCT "<PC>".
 
Review the monitoring agent's RAS1 log to confirm the IP address for the connection to the TEMS, and whether Self-Describing Agent functionality is enabled:
   kraaulog.cpp,755,"IRA_OutputLogMsg") Connecting to CMS <TEMS_name>  
   kraarreg.cpp,1125,"ConnectToProxy") Successfully connected to CMS  <TEMS_name> using ip.pipe:#111.222.333.444 [1918]  
   kraaulog.cpp,755,"IRA_OutputLogMsg") Self-Describing Agent function disabled by TEMS "<TEMS_name>" for PRODUCT "<PC>".  
 
Review the configuration settings and RAS1 logs for the TEMS the agent indicates it is connecting to:
In the TEMS RAS1 log, look for an indication of the "Node Status Ignored" message:
   kfastinh.c,1087,"KFA_InsertNodestatus") Affinity not loaded for node <system:<PC>> thrunode <TEMS_name> affinities <xxx>.  Node Status Ignored.  
   kpxreqhb.cpp,904,"HeartbeatInserter") Remote node <system:<PC>> is ON-LINE.  
   kpxreqhb.cpp,899,"HeartbeatInserter") Remote node <system:<PC>> is OFF-LINE.  
 

If the reason that the endpoint is not on-line is due to the node status being ignored, confirm the application support installed on TEMS system(s).
 
For Windows, review "kincinfo -d" output.
For UNIX / Linux, review "cinfo -i" output.
 
Or review the "cinfo.info" included in "pdcollect" output file.
 
"Node Status Ignored" is an indication that application support for the agent type - <PC> - is not installed on the TEMS.  
   

Resolution
If the reason for the agent on the endpoint being off-line is due to missing application support on the HUB TEMS or RTEMS, install application support for that agent.
 
This can be done by manually installing application support from the agent install image, or if the Tivoli Monitoring components support Self-Describing Agent functionality, by correcting what is preventing SDA.
 
Reasons that can prevent SDA from working.  
 
Confirm that "KMS_SDA=Y" is set on the HUB TEMS and RTEMS.
 
By default, self-describing is disabled at the hub monitoring server.  
Self-describing is enabled, by default, for all remote monitoring servers (RTEMS), the portal server (TEPS), and agents enabled for self description.  
KMS_SDA=Y environment variable must be set on each hub monitoring server in the environment.  
 
The agent must also be enabled for self-description by packaging application support as part of the agent installation.
The agent must use the IBM Tivoli Monitoring V6.2.3 or later monitoring agent framework or be installed on a  system where the V6.2.3 or later agent framework is already installed.  
 
At 6.3 release, additional changes were made regarding SDA functionality that  need to be configured on the HUB TEMS.
 
Please refer to the section "Managing your self-describing agent installations"  
http://www-01.ibm.com/support/knowledgecenter/SSTFXA_6.3.0/com.ibm.itm.doc_6.3/install/manage_sda.htm?lang=en  
***
As of IBM® Tivoli® Monitoring V6.3, the hub monitoring server does not allow the automatic installation of a self-describing agent enabled product.  
You must first specify the products and versions that can be installed, or change the default self-describing agent installation behavior to permit  
installations for all products and versions.  
To allow default SDA enablement:  
1. Ensure that there are no products configured for self-describing agent installation by issuing the command  
   tacmd deletesdainstalloptions -all  
Attention: This will remove any previous self-describing agent product versions and configurations that were added.  
2. Issue the command  
   tacmd editsdainstalloptions -t DEFAULT -i ON  
***
 
Prior to making changes to "enable" the SDA by default with the commands above, gather the following from the HUB TEMS system:  
  tacmd listSdaInstallOptions  
  tacmd listSdaOptions 

Prior to 6.3 release of ITM, you could enable or disable SDA at the TEMS with "KMS_SDA=" parameter.
This either completely enabled SDA, or completely disabled SDA function.

Some customers wanted the ability to control SDA on a more granular level where they could allow SDA for specific types of agents, but disable it for others, to prevent problems where SDA unexpectedly might upgrade application support.

Starting with 6.3 release of ITM, there is an extra step that must be completed in addition to setting KMS_SDA=Y, to explicitly allow SDA on a per application basis.
Explicitly enable SDA for a given application, or enable SDA globally as the default.

There are multiple things that can prevent SDA from the ITM infrastructure side in 6.3.x environments

1) The agent has to be SDA capable (meaning that it has high enough TEMA common components to support SDA).
2) The agent image has to ship the application support definitions so they can be pushed to the TEMS upon connection, and the details need to be in the  K<pc>MSMAN.txt file;
   The agent image must contain the "K<pc>JSTMS.jar" / "K<pc>JSTPS.jar" / "K<pc>JSTPW.jar" / "K<pc>MSMAN.txt" / "k<pc>_sda_<version>.jar" files.
3) The agent environment file has to have the setting to allow for SDA:
   K<pc>ENV / <pc>.ini must have:
  TEMA_SDA=Y
4) The TEMS the agent connects to must be of high enough level to support SDA
    SDA function was introduced starting in 6.23 release.
5) The TEMS must specify:
   TEMS_SDA=Y
   (This defaults to enabled on RTEMS, but is disabled by default on HUB TEMS)
6) If the envorinment is 6.3 or higher, the HUB TEMS must take additional steps to enable SDA in addition to #5
7) The TEPS must have SDA enabled, in <candle install>/config/cq.ini
    TEPS_SDA=Y
8) TEPD does not support SDA, application support must manually be installed on TEP Desktop systems that are running separately from the TEPS

Submitter: drd401709
Reference DCF technotes:
DCF 1698039 - Tivoli Monitoring agent is off-line even though it successfully connects to TEMS

Keywords:

offline