IBM APM 8.1 - Synthetic agent SN - Useful info - tips - Part 2 of 3

Body

Environment:

IBM Cloud Application Performance Management Server, V8.1.4 IF7

IBM Cloud Application Performance Management advanced, Synthetic agent, V8.1.4 IF3

Selenium scripts (.html and .zip)

Problem:

After you have recorded your Selenium test scripts (using Selenium IDE plug-in), uploaded them to the APM Server, and then deployed them to a Synthetic SN agent, using APM UI Synthetic Script Manager, you notice that some transactions show  :

'Latest status' field  =  grey "?", meaning unknown.

If the observed phenomenon is that that some transactions can be played back, and some can not, and after SN agent is restarted, some others/different ones may have been played back but some others may not have, it looks like the SN agent does not have enough capacity or resources to run more or even all the currently scheduled playbacks.

The best approach then is to edit the latest ksn_trace*.log log files (e.g /opt/ibm/apm/agent/logs/ksn_trace0.log)  and search for the key word :

threadPoolExecute queue size

What is the queue size value mostly seen in this log file?

If the queue size value is always high, that means that the scheduled playback tasks can not be completed in time, and those playback tasks will be accumulated. 

This would explain that some Transactions appear in APM UI under 'Synthetic Transactions' widget with 'Latest status' = unknown

Example:

[2018-12-10T12:27:23.120-0400] - FINEST - xxxxxxx- Thread-41 - com.ibm.csi.apm.synthetics.playback.Playback.startScriptRunnable() - threadPoolExecute queue size :51448 task count : 81022

here above, 51448 playback tasks heap up, so any new added script can not get its turn to playback

Solution:

If you determine that the workload is too heavy for this agent, we can consider several approaches :
1) deploy another SN agent and distribute some scripts into that new synthetic agent, to share the load between several agents
2) increase the playback interval (for example from 2 minutes to 5 minutes)

3) increase the capacity (RAM, CPU) of the machine hosting the SN agent

A first approach can consist simply in increasing the current playback interval of all transactions, then restart the SN agent, and verify if everything is now OK. Check again in ksn_trace*.log log file if the queue size keeps increasing. If its value is larger than 1000, then you need to consider to increase again the playback interval, or consider next step like adding more capacity to the SN agent host machine.