IBM Support

APM V8 Transactions are intermittently missing in the dashboard

Technical Blog Post


Abstract

APM V8 Transactions are intermittently missing in the dashboard

Body

Product

IBM Cloud Application Performance Management, Private

Scenario

On APM V8 dashboard, transactions data for various agents shows gaps even though the transactions continue executing during that period.

Consider the scenario where you have Synthetic Playback transactions running every 2 minutes. The Synthetic Playback agent logs show that the playbacks are running every two minutes without errors, however the transactions on dashboard show gaps.

For example, you can see transactions with timestamp July 13,2018, 10:23 and then at July 13,2018, 10:33. No transactions show at 10:25, 10:27 etc.

What can be causing the transactions data to go missing?

Cause

To protect APM server from resource crunch due to excessive transaction records (also known as Application Activity Records or AARs), there is a setting to limit the AAR (transactions) data processed by APM server. If this limit is exceeded due to high transactions rate, then the processing of AAR data is halted for a period. When this happens, the Transactions data will be missing on the dashboard during these periods, as AAR processing was halted during that period.

Diagnosis

To check if you have encountered this issue, check 'min' server logs on APM server host:

cd <install_dir>/wlp/usr/servers/min/logs
grep KafkaMonitor messages*

You may find messages like the following:
com.ibm.tivoli.ccm.ingress.rest.KafkaMonitor                 E HALT state is set for IngressRest.   The inbound AAR data rate is too high and the fifteen minute rate has been exceeded.  Current fifteen minute rate: 313.3799298150302.  Configured rate: 300.0

com.ibm.tivoli.ccm.ingress.rest.KafkaMonitor                 E IngressRest has been HALTed for 30005 milliseconds.

Resolution

Configured rate: 300.0 in the KafkaMonitor log message refers to setting in following file:
 <install_dir>/wlp/usr/servers/min/bootstrap.properties
 com.ibm.tivoli.ccm.jmxthreshold=300
 

The values set by the server_size.sh script for the jmxthreshold are:  300 for small environment,  750 for medium, and 3000 for large. 

This value can also be set either by running server_size.sh script or manually by editing the file:
     <install_dir>/wlp/usr/servers/min/bootstrap.properties

and setting the following variable:

     com.ibm.tivoli.ccm.jmxthreshold=x

where x is a value suited for the rate of transactions.  Refer to next section for setting correct server size before increasing this threshold.

Important notes:

If you increase AAR threshold, then you will need to correspondingly increase the hardware resources and other APM server settings to avoid running into resource issues for the APM server.

  • The APM 8.1.4 hardware requirements table for the Advanced offering is shown on Table 2 at this link:

            https://www.ibm.com/support/knowledgecenter/SSHLNR_8.1.4/com.ibm.pm.doc/install/planning_requirements_serverhwreqs.htm

            The memory requirements for the server1, apmui and spark processes grow with an increasing AAR rate. 

            The recommended CPU and Memory sizes shown in Table 2 were chosen based on experience in lab environments.

  • Refer to following blog entry for APM server size.

            Setting the appropriate APM server size
             https://www.ibm.com/developerworks/community/blogs/0587adbc-8477-431f-8c68-9226adea11ed/entry/APM_V8_1_4_Planning_installation_and_setting_correct_Server_Size

 

 

 

Tutorials Point

 

Subscribe and follow us for all the latest information directly on your social feeds:

 

 

image

 

image

 

image

 

 

  

Check out all our other posts and updates:

Academy Blogs:https://goo.gl/U7cYYY
Academy Videos:https://goo.gl/TLfMoF
Academy Google+:https://goo.gl/HnTs0w
Academy Twitter :https://goo.gl/AhR8CL


image

[{"Business Unit":{"code":"BU004","label":"Hybrid Cloud"},"Product":{"code":"","label":""},"Component":"","Platform":[{"code":"","label":""}],"Version":"","Edition":""}]

UID

ibm11082907