Technical Blog Post
APM V8 Transactions are intermittently missing in the dashboard
IBM Cloud Application Performance Management, Private
On APM V8 dashboard, transactions data for various agents shows gaps even though the transactions continue executing during that period.
Consider the scenario where you have Synthetic Playback transactions running every 2 minutes. The Synthetic Playback agent logs show that the playbacks are running every two minutes without errors, however the transactions on dashboard show gaps.
For example, you can see transactions with timestamp July 13,2018, 10:23 and then at July 13,2018, 10:33. No transactions show at 10:25, 10:27 etc.
What can be causing the transactions data to go missing?
To protect APM server from resource crunch due to excessive transaction records (also known as Application Activity Records or AARs), there is a setting to limit the AAR (transactions) data processed by APM server. If this limit is exceeded due to high transactions rate, then the processing of AAR data is halted for a period. When this happens, the Transactions data will be missing on the dashboard during these periods, as AAR processing was halted during that period.
To check if you have encountered this issue, check 'min' server logs on APM server host:
grep KafkaMonitor messages*
You may find messages like the following:
com.ibm.tivoli.ccm.ingress.rest.KafkaMonitor E HALT state is set for IngressRest. The inbound AAR data rate is too high and the fifteen minute rate has been exceeded. Current fifteen minute rate: 313.3799298150302. Configured rate: 300.0
com.ibm.tivoli.ccm.ingress.rest.KafkaMonitor E IngressRest has been HALTed for 30005 milliseconds.
Configured rate: 300.0 in the KafkaMonitor log message refers to setting in following file:
The values set by the server_size.sh script for the jmxthreshold are: 300 for small environment, 750 for medium, and 3000 for large.
This value can also be set either by running server_size.sh script or manually by editing the file:
and setting the following variable:
where x is a value suited for the rate of transactions. Refer to next section for setting correct server size before increasing this threshold.
If you increase AAR threshold, then you will need to correspondingly increase the hardware resources and other APM server settings to avoid running into resource issues for the APM server.
- The APM 8.1.4 hardware requirements table for the Advanced offering is shown on Table 2 at this link:
The memory requirements for the server1, apmui and spark processes grow with an increasing AAR rate.
The recommended CPU and Memory sizes shown in Table 2 were chosen based on experience in lab environments.
- Refer to following blog entry for APM server size.
Setting the appropriate APM server size
Subscribe and follow us for all the latest information directly on your social feeds:
|Academy Twitter :||https://goo.gl/AhR8CL|