Verifying your Thresholds have been distributed to your monitored Agents.

Body

If you've created few Thresholds in your APM 8.1.3 console and you're wondering if these Thresholds have been distributed to your Agent system, here are some steps that you can use to verify:

1. When you create your Threshold in the APM console using the Threshold Manager, the new Threshold that you created gets written into a private_situations.xml file on the APM server.  This is a regular xml file that contains all your Thresholds and is in the wlp servers directory (see screenshot below for the path.)

In my example here, I've created two Thresholds: NL_LZ_Disk_Usage (to check disk usage over a certain value) and NL_LZ_Process_CPU_Check (to check if a particular process is using high CPU) using the Threshold Manager and both these Thresholds get written into the private_situations.xml file.

2. So if you look at the private_situations.xml file on the APM server in this dir, you will find two entries, one for each Threshold.  See the screenshot below showing the dir path and the two Threshold entries I created.  Click on the screenshot directly should you need a bigger view.

3. Now that the APM server has these entries, it should push out the private_situations.xml file to your distributed Agents.  The distributed Agents are the ones in your distribution Group from the Threshold definition.   So the next time the monitored Agent checks in with the APM server in 5 minutes, the server will push out its private_situations.xml file to the agent system.   If you logon to the agent system and go to the dir shown in the screenshot below (in this example /opt/ibm/apm/agent/localconfig/lz) you will find that the agent's system now has an updated private_situations.xml file with the above two Threshold entries.  It looks like this, click on the screenshot directly if you need to see a bigger view:

These are the basic steps you should check to see if your Thresholds have been distributed successfully from your APM server to the agents.  If you find that your Agent system's private_situations.xml file does not have the updated Thresholds in it, wait about 5 minutes and re-check.  If you still don't find it, then go back to your APM server, make a simple change to your Threshold from the Threshold Manager or create a new one, and see if the private_situations.xml file on the server gets updated first.  Then the next time the monitored Agent checks in with the APM server, the file should get pushed out.   If it still doesn't get updated, then check the distribution group you have defined in the Threshold.  If you are using a Custom group, the Agent system must be in the Custom group.

If you are running into problems and still don't see the Agent's private_situations.xml file getting updated, then you will need to collect the following information and open a pmr with IBM Support. Please collect this info and send it in for review with your new pmr.

•     collectLogs.sh from your APM server
•     pdcollect from the distributed Agent system
•     screenshot from your Threshold Manager showing the Threshold you've created
•     if using a Custom group, capture a screenshot of this Custom group from the Resource Group Manager screen showing the distributed agent(s)

Note: There have been few fixes for the Threshold definitions (specifically, APAR IV91578 which is in IF08).  So, its best to put on the latest patch (IF09 at the time of writing this blog) on the server side.  Installation of the server side patch is very easy but a restart of your APM server is required.

Good luck.
Noel Lewis

ITCAM / APM / ICAM L2 Support team

Subscribe and follow us for all the latest information directly on your social feeds: