Self-hosted PoP Capacity planning and scaling

Capacity planning

You might want to know, how to plan the capacity of Self-hosted Synthetic PoP deployment, and how to scale up PoP to support an increasing workload. The PoP size and resource depend on the workload, such as the number of Synthetic tests, the test execution frequency, and the test script complexity. A good capacity plan can provide you with better performance and stability at the lowest cost. Instana provides a reference for different PoP size and resource allocations based on measurements and projections from standard IBM benchmarks in a controlled environment. Actual throughput or performance varies due to many factors, follow these steps to improve PoP performance:

• Planning enough resource allocation based on workload and PoP size estimation.
• Monitoring PoP health status with Instana Agent in a production environment.
• Scaling up specific playback engines on demand to support an increasing workload.

If you use the default CPU and memory resource allocation for a Synthetic PoP, ensure that your Kubernetes cluster has at least a 6 core CPU and 4.8 GB memory available in its physical resources, then you can run 2000 API Simple tests, 100 API Script tests, 15 Browser Script tests, and 2000 ISM tests, based on the performance benchmark testing with the following test configuration:

To install the Instana Agent to monitor your PoP or to scale up your PoP to improve the performance, more physical resources must be available in the Kubernetes cluster.

The estimation of PoP size and resource allocation is done by testing with the constant Synthetic test and the test frequency as shown in the table, using the resource requests and limits from each component's default configuration. For information on the default configuration of Synthetic PoP components, see PoP helm charts. Similarly, for information on the default configuration of the Instana Agent and Kubernetes (K8) sensor, see Agent helm charts. The number of playback engine replicas is calculated based on the number of tests and their frequency of execution. The number of Instana agents is calculated based on the number of Kubernetes worker nodes. In the default configuration, the number of Kubernetes sensors is set to 3. With 3 replicas of Kubernetes sensors, you can monitor up to 400 nodes.

Estimate PoP size with automation tool

Instana provides an automation tool synctl to help you estimate PoP size and plan capacity. You can use the tool as shown in the following examples:

synctl get pop-size

synctl get size

Scaling up

To support increasing workloads, scale up PoP components as follows:

• Horizontal scaling: a simpler approach is provided to increase the replica number of different playback engines.
• Vertical scaling: increase the requests and limits numbers of CPU and memory.

--set javascript.replicas=2
--set browserscript.replicas=3

Performance tuning

To support workloads above 2000 Synthetic tests per minute, tune the PoP controller as follows:

• To support more tests with user credentials, tune the controller.scheduleTestMaxPoolSize parameter to a higher value. To support this parameter, upgrade to Synthetic PoP 1.1.2 or later.
• To support more results that are sent to the Instana backend, tune the controller.publishAARThreadPoolSize parameter to a higher value.

If you tune the parameters controller.resources.limits.cpu to 500 m, controller.resources.limits.memory to 500 Mi, controller.publishAARThreadPoolSize to 40, and controller.scheduleTestMaxPoolSize to 15, the PoP controller can support 9000 API Simple tests with user credentials per minute.