APM v8.1.4 Setting the appropriate APM server size

Technical Blog Post

Abstract

Body

Lack of sufficient hardware resources and java heap and other settings will cause unexpected behavior in the APM monitoring environment, such as agent not showing up in the dashboard, data not showing in the dashboard, APM server processes not starting and so on. Therefore it is very important to set the correct APM server size, and also to provision recommended hardware resources on the APM server host.

1) Environment

This entry applies to APM server component of following product. It will be referred to as "APM Server" in this discussion.

IBM Cloud Application Performance Management, Private (v8.1.4)

2) Topic of discussion

Hardware resources and APM server(s) configuration settings which are required for optimal functioning of APM server, increase with the number of managed systems (agents).

APM provides three server size options based on the load on the APM server, estimated based on the of number of agents, data rows per minute per agent and transactions per second.

We will discuss following topics:

- Concept of APM server size

- Using server_size.sh script to check and set APM server size

Note: For this discussion, we will assume default install path of /opt/ibm for the APM server.

3) What are APM server size settings?

APM Server size not a property setting, it is a collection of configuration settings (for server processes, database servers), and hardware requirements for the APM server host.

APM Server size is set by the installer as "small", "medium" or "large" depending on available memory and CPUs.

Installer also configures the Java heap size settings of APM server processes according to the size setting. In the installer log /opt/ibm/ccm/logs/apm-server-install*.log, following server size log entry can be found:

Wed Jan 31 16:21:06 EST 2018: The server size has been configured as 'small' based on the number of CPUs and amount of memory. To reconfigure the server size, run script /opt/ibm/ccm/server_size.sh with the desired size as a parameter. Valid sizes are: small, medium, large.
Wed Jan 31 16:21:06 EST 2018: Please review the documentation at http://ibm.biz/mon_doc for more information.

This hardware requirement of APM server itself is based on the number of managed systems (aka agents or resources) connecting to the APM server.

The list of hardware required for "small", "medium" or "large" APM server sizes, is published on the following Knowledge Center page. Table 1 is for APM base offering and Table 2 is for APM advanced offerings; Table 3 lists the Java Heap size for various servers such as apmui, server1 etc. that make up the APM server.:

https://www.ibm.com/support/knowledgecenter/SSHLNR_8.1.4/com.ibm.pm.doc/install/planning_requirements_serverhwreqs.htm

The hardware recommendations in these tables assume baseline of number of rows of monitoring data per agent and transactions per minute as noted in rows 2 and 3 of tables 1 and 2. Some agents may send more rows per minute than some other agents, and transactions per second also vary in different environments. Thus the settings in these tables are to be used as a guideline only, your requirement of APM server hardware may be greater or smaller based on the type of agents and the workload. See References section at the end of this post for additional information.

4) Using server_size.sh script

APM provides server_size.sh script which is located in /opt/ibm/ccm directory. This script can be used to find the recommended and current size setting and also to change the server size settings.

Checking APM server size setting:

When you run the script without any parms, the script checks for the RAM and CPUs and puts out the recommended and current server size.

ccm]# ./server_size.sh
Server size does not meet minimum requirements for number of CPUs or size of RAM. Defaulting to small.
Number of CPUs: 6
Memory size: 20.00 GB
Recommended server size: small
Current server size: small

5) Determining required APM server size

In order to check if this is a good server size for your environment, it is best to first determine the total number of agents that you eventually plan to connect to the APM server. If you have inherited a system, and want to check the current number of online agents, you can use steps in the following section:

How may agents are connected?

Check the number of online agents by running following commands on the APM server host (replace apmhost with the ipaddress of host name of the APM server):

# curl http://apmhost/1.0/monitoring/agent/inventory -H "Accept: text/plain">agents.log

# grep " online" agents.log | wc -l

1500

This shows that you have 1500 "online" agents, which requires APM Server size of medium (see tables at KC link in section 3). The server_size.sh shows that the current and recommended server size settings are "small".

6) Steps to increase the server size

Now you know the following:

1. Total number of agents and therefore the desired size of the APM Server.

2. Current and recommended size of APM server as reported by server_size.sh script.

Next step is to look up the hardware requirement for the desired APM Server size (KC link in section 3), this includes the number of processors, CPUs, memory (RAM), disk space, HDD count, Disk I/O TPS. Increase the hardware accordingly and then run server_size.sh script to configure the desired APM Server size. One run of the APM server size setting to small is given below.

Following is example run of the "small" server size setup, the settings will be different for "medium" and "large" sizes. Note that the APM server processes are restarted as a result.

ccm] # ./server_size.sh small
Setting heap size to 1526 MB for service apmui.
Setting heap size to 256 MB for service kafka.
Setting heap size to 512 MB for service min.
Setting heap size to 128 MB for service oidc.
Setting heap size to 1526 MB for service server1.
Setting heap size to 1024 MB for service uviews.
Setting heap size to 256 MB for service zookeeper.
Setting ENABLE_MY_COMPONENTS to true for service apmui.
Setting JMX threshold to 300 for service min.
Setting prefetch pause to 100000 for service min.
Setting prefetch resume to 75000 for service min.
Setting cache size (GB) to 1 for service mongodb.
Setting datamart_ibm_default_buf_pool value to 100000.