APM V8 Transactions are intermittently missing in the dashboard

Body

Product

IBM Cloud Application Performance Management, Private

Scenario

On APM V8 dashboard, transactions data for various agents shows gaps even though the transactions continue executing during that period.

Consider the scenario where you have Synthetic Playback transactions running every 2 minutes. The Synthetic Playback agent logs show that the playbacks are running every two minutes without errors, however the transactions on dashboard show gaps.

For example, you can see transactions with timestamp July 13,2018, 10:23 and then at July 13,2018, 10:33. No transactions show at 10:25, 10:27 etc.

What can be causing the transactions data to go missing?

Cause

To protect APM server from resource crunch due to excessive transaction records (also known as Application Activity Records or AARs), there is a setting to limit the AAR (transactions) data processed by APM server. If this limit is exceeded due to high transactions rate, then the processing of AAR data is halted for a period. When this happens, the Transactions data will be missing on the dashboard during these periods, as AAR processing was halted during that period.

Diagnosis

To check if you have encountered this issue, check 'min' server logs on APM server host:

cd <install_dir>/wlp/usr/servers/min/logs
grep KafkaMonitor messages*

You may find messages like the following:
com.ibm.tivoli.ccm.ingress.rest.KafkaMonitor                 E HALT state is set for IngressRest.   The inbound AAR data rate is too high and the fifteen minute rate has been exceeded.  Current fifteen minute rate: 313.3799298150302.  Configured rate: 300.0

com.ibm.tivoli.ccm.ingress.rest.KafkaMonitor                 E IngressRest has been HALTed for 30005 milliseconds.

Here the Configured rate: 300.0 in the KafkaMonitor log is the AAR threshold setting in the min server configuration.

Resolution

To resolve this problem, increase the AAR threshold setting com.ibm.tivoli.ccm.jmxthreshold in the following file:
 <install_dir>/wlp/usr/servers/min/bootstrap.properties
 com.ibm.tivoli.ccm.jmxthreshold=300

The AAR threshold setting ( com.ibm.tivoli.ccm.jmxthreshold) is 300 for a small environment,  750 for a medium environment, and 3000 for a large environment. This is set when you run server_size.sh script.

You can change this property by running server_size.sh script or by manually editing the properties file.
Manually edit <install_dir>/wlp/usr/servers/min/bootstrap.properties file and change the following  property to a value <number> suited for the rate of transactions.

     com.ibm.tivoli.ccm.jmxthreshold=<number>

Note that if you decide to increase the AAR threshold setting, then you will need to correspondingly increase the hardware resources and other APM server settings to avoid running into resource issues for the APM server. After allocating sufficient hardware resources, you must run server_size.sh script to increase the size of APM server.

• The APM 8.1.4 hardware requirements table for the Advanced offering is shown on Table 2 on the following page:

            Cloud APM server hardware requirements

            The memory requirements for the server1, apmui and spark processes grow with an increasing AAR rate. 

            The recommended CPU and Memory sizes shown in Table 2 were chosen based on experience in lab environments.

• Refer to following blog entry for APM server size.

            Setting the appropriate APM server size