Synthetic PoP Deployment

Synthetic PoP (Point of presence) is the agent where the synthetic tests are run. You can deploy Synthetic PoP in multiple locations to monitor your application.


Synthetic PoP runs as a set of kubernetes pods in the kubernetes cluster, see below architecture.

Note: Only two playback engines are supported: the HTTP engine that runs the HTTPAction test and the JavaScript engine that runs the HTTPScript test. More playback engines, such as browser script and ISM, will be supported in later releases.

Deployment Option

Synthetic PoP can be deployed only by the Helm charts.

  • The official Helm repository URL is, and the chart name is synthetic-pop.
  • The Helm chart git repo is


Synthetic PoP needs to be deployed in a Kubernetes cluster. Use the helm chart to deploy Synthetic PoP, and ensure that Kubernetes and helm are installed in your environment.

  • Set up a Kubernetes cluster where the PoP will be installed. The minimum supported version for the kubernetes distribution is 1.10. For more information, see Set up Kubernetes.
  • Install Helm, and Helm V3 is recommended. For more information, see Installing Helm.

The minimum physical requirement for the kubernetes cluster is 2 Core CPU and 2GB memory.


Helm Install

Use the helm install command to install a new Synthetic PoP, and pass the following values. You are suggested to install the PoP in a separate namespace.

  • downloadKey: The download agent key for pulling image from Docker registry.
  • controller.location: The format is label; displayLabel; country; city; latitude; longitude; description. This label acts as the PoP identifier and can only contain letter, number, hyphen and underscore.
  • controller.instanaKey: The Instana agent key, which is used for connecting to Instana backend.
  • controller.instanaSyntheticEndpoint: The ingress endpoint of synthetic acceptor in Instana backend.
  • controller.clusterName: Optionally. It needs to be set with the name of the Kubernetes cluster that is to be displayed in the Instana UI when Instana agent is installed to monitor this PoP.
  • redis.password: The authentication password to redis server. Specify a password with length at least 10.
  • redis.tls.enabled: Define if enabling Redis TLS or not (default false). If the value is true, the communication with Redis will be encrypted with TLS.
  • redis.tls.secretName: If the value of the redis.tls.enabled field is true, you need to specify a secret name for Redis TLS key/cert files.

Sign in to Instana UI, click Synthetic Monitoring, and then click Deploy a PoP. Copy the values of downloadKey, controller.instanaKey and controller.instanaSyntheticEndpoint.

To configure the installation, specify the values in the command line by using the --set flag, or provide a yaml file with your values by using the -f flag.

In the following example, a PoP is deployed, and Redis uses password for authentication.

helm install synthetic-pop \
    --repo  \
    --namespace syn \
    --create-namespace \
    --set downloadKey="yourdownloadkey" \
    --set controller.location="MyPoP;My PoP;China;Beijing;39.54;116.23;This is a testing Synthetic Point of Presence" \
    --set controller.instanaKey="instanaAgentkey" \
    --set controller.instanaSyntheticEndpoint="" \
    --set redis.tls.enabled=false \
    --set redis.password="a1fc5d01bcbb" \

You can also provide your values.yaml to deploy a PoP, below examples shows how to get the values.yaml template and then pass the modified yaml file to deploy a PoP.

helm repo add synthetic-pop-repo
helm repo update
helm pull synthetic-pop-repo/synthetic-pop
tar xzvf synthetic-pop-*.tgz
# Modify the synthetic-pop/values.yaml file
# Run helm install to pass the modified values.yaml
 helm install synthetic-pop \
    --repo  \
    --namespace syn \
    --create-namespace \
    -f /path/to/values.yaml \

After PoP is installed, make sure that each pod is in Running and Ready status.

kubectl get pod -n syn

Sign in to Instana UI, click Synthetic _Monitoring > Locations, and then you can see your location is showed up in the location list. If not, go to the Troubleshooting section to check the log files.

Security Enhancement

The following configurations are recommended for security enhancement in production environment.

Redis TLS

If you want to enable Redis TLS, you need to have your X.509 certificate-key pair (tls.crt, tls.key) and CA root certificate file (ca.crt).

If you do not have your key/cert files, you can also use openssl command to generate new ones. See the following examples:

openssl genrsa -out tls.key 4096
openssl req -x509 -new -nodes -sha256 -key tls.key -days 3650 -subj '/O=Redis/CN=Certificate Authority' -out ca.crt
openssl req -new -sha256 -key tls.key -subj '/O=Redis/CN=Server' | openssl x509 -req -sha256 -CA ca.crt -CAkey tls.key -CAserial ca.txt -CAcreateserial -days 3650 -out tls.crt

After key/cert files are provided, use the kubectl command to create a secret.

# Create a new TLS secret named redis-tls-secret with the given key/cert files
kubectl create secret generic redis-tls-secret -n syn \
--type='' \
--from-file=ca.crt=path/to/ca.crt \
--from-file=tls.crt=path/to/tls.crt \

Use the following installation command to deploy a PoP to support Redis TLS:

helm install synthetic-pop \
    --repo \
    --namespace syn \
    --create-namespace \
    --set downloadKey="yourdownloadkey" \
    --set controller.location="MyPoP;My PoP;China;Beijing;39.54;116.23;This is a testing Synthetic Point of Presence" \
    --set controller.instanaKey="instanaAgentkey" \
    --set controller.instanaSyntheticEndpoint="" \
    --set redis.tls.enabled=true \
    --set redis.tls.secretName="redis-tls-secret" \

Secure Computing Mode

You can use secure computing mode (or seccomp) to sandbox the privileges of a process, which restricts the calls from user space into the kernel. Use the following parameter to enable RuntimeDefault as default seccomp profile for all containers.

Note: The default seccomp profile works only for Kubernetes version 1.19 or later or OCP (OpenShift Container Platform) version 4.11 or later. If seccompDefault is not set to true for OCP 4.11 or later, you can see some warning messages when you run the helm installation.

helm install synthetic-pop \
    --repo \
    --namespace syn \
    --set seccompDefault=true \

Network Policy

Network policy is installed by default. To block access from PoP to specific IPs or IP ranges, you can customize the playbackEngineNetworkPolicy parameter. Typically, you can collect the following IP addresses:

  • Cloud provider metadata service IP address: It is for AWS Metadata API, Google Cloud Metadata API, Azure Metadata API.
  • Kubernetes API server IP address: Use the kubectl command to collect Kubernetes API server IP address. See the following example for microk8s or minikube Kubernetes cluster:
# Use the following command to print apiserver endpoint
# For example, if the following command prints "Endpoints:", you need to block the IP address 
kubectl describe service kubernetes

Then, modify the file values.yaml in helm charts as follows to enable Egress rules to block the IP addresses that you collect.

  # Change the value to true to enable the network policy
  enabled: true
    - to:
        - ipBlock:
            # Allow all IP address v4
              # Block Kubernetes API server IP address
              # Block cloud provider metadata service IP address

See the following example to pass network policy by using --set:

# --set-json only be supported since Helm V3.11
helm install synthetic-pop \
    --repo \
    --namespace syn \
    --set playbackEngineNetworkPolicy.enabled=true \
    --set-json 'playbackEngineNetworkPolicy.egress=[{"to":[{"ipBlock":{"cidr":"","except":["",""]}}]}]' \

Migration Scenario

To support the backend migration scenario, you can specify multiple synthetic acceptor endpoints and Instana agent keys with semicolon. Instana agent keys and synthetic acceptor endpoints are one-to-one correspondence. During the migration period, Synthetic PoP can send data to both old and new backend to avoid downtime.

See the following example:

helm install synthetic-pop \
    --repo \
    --namespace syn \
    --set controller.instanaKey="instanaAgentkey;anotherInstanaAgentKey" \
    --set controller.instanaSyntheticEndpoint=";" \


Use the helm upgrade command to update a Synthetic PoP. See the following example. You can change the parameters as needed.

helm upgrade synthetic-pop \
    --repo \
    --namespace syn \
    --set downloadKey="yourdownloadkey" \
    --set controller.location="MyPoP;My PoP;China;Beijing;39.54;116.23;This is a testing Synthetic Point of Presence" \
    --set controller.instanaKey="instanaAgentkey" \
    --set controller.instanaSyntheticEndpoint="" \
    --set redis.tls.enabled=false \
    --set redis.password="a1fc5d01bcbb" \


  • The label value in the location parameter is an identifier of the PoP, and not allowed to change. You are recommended to change the displayLabel value as you want.
  • The redis password is not allowed to update. If you really need to change it, uninstall it and then reinstall it with updated redis password.


Use the helm uninstall command to uninstall a Synthetic PoP.

helm uninstall synthetic-pop -n syn

Note: After the PoP is uninstalled, the PoP's location is still shown in the Instana UI with offline status. You can reinstall the PoP with the same location to make the location online again, or delete the location on UI if you do not use it anymore.

More Configuration Options


Currently, only playback engines support horizontal scalability, and PoP Controller and Redis don't support horizontal scalability yet. The PoP workload is mainly on the playback engine pods. So to support more workload on a certain type of test, just increase the replicas number in the values.yaml file.

The following example shows that the replicas of HTTP playback engine are increased to 2 and replicas of javascript playback engine are increased to 3.

--set http.replicas=2
--set javascript.replicas=3


The controller.capabilities property defines the playback capabilities for this PoP. Only the HTTPAction and HTTPScript test types are supported.


The controller.customProperties property defines the customized tags or properties for this PoP. The property is a list of key:value pairs, separated by a semicolon, such as key1=value1;key2=value2.

Use an API such as $env.key1 to access the custom property or tag in an API or browser script.


You can configure a proxy server for synthetic PoP to connect the Instana backend server. The following options are supported.

  • proxy.popProxyProtocol: the protocol used by the proxy server. Set the value to http or socks.
  • proxy.popProxyServer: the proxy server address in the ipaddress:port format.
  • proxy.popProxyUserPass: the username and password to authenticate the proxy server. Enter the username and password in the username:password format.

Using external secrets

If you don't want to use Helm chart to create new secrets for the keys controller.instanaKey, redis.password, proxy.popProxyUserPass, and downloadKey, you can use the existing secrets that you have created for the Synthetic PoP deployment by specifying the secret names as follows:

helm install synthetic-pop \
    --repo \
    --namespace syn \
    --set downloadSecret="my-pull-secret" \
    --set controller.instanaKeySecret="my-pop-secret" \
    --set redis.passwordSecret="my-pop-secret" \ 
    --set proxy.popProxyUserPassSecret="my-pop-secret" \

For the keys controller.instanaKey, redis.password, and proxy.popProxyUserPass, you can use the same secret or different existing secrets. However, when you create the secret, you must use the specified key fields instana-key, redis-password, and proxy-user-pass as follows:

 kubectl create secret generic my-pop-secret  --namespace <namespace> \
 --from-literal=instana-key='<instana agent key>'  \
 --from-literal=redis-password='<redis password>'  \
 --from-literal=proxy-user-pass='<proxy username and password>'  

Note: You need to replace <namespace> with the namespace that you use to install the Synthetic PoP.

The secret for the downloadKey key must use the type. You can create the secret with the type as follows:

kubectl create secret docker-registry my-pull-secret --namespace <namespace> --docker-server="" --docker-username="_" --docker-password="<downloadKey>" 


  • Replace <namespace> with the namespace that you use to install the Synthetic PoP.
  • Don't change the value of the --docker-username argument.
  • Replace <downloadKey> with the Instana download key.


To check the Helm chart version, run the following command:

helm list -n <namespace>

You can get the version number from the result such as synthetic-pop-<version>.

If you meet any issues, for example, the location that you specify can't be shown on Instana Synthetic UI, first check the PoP controller log by running the following command:

kubectl logs -n <namespace> -f <pop_controller_pod_name>

To change the trace level of each component, you can pass the traceLevel values as follows to make an upgrade on your PoP:

--set controller.traceLevel="DEBUG"
--set http.traceLevel="DEBUG"
--set javascript.traceLevel="DEBUG"

You can also dynamically change the trace level of PoP controller without restarting the pod by running the command in the container.

# enter synthetic-pop-controller container
kubectl exec -it -n <namespace> <pop_controller_pod_name> -- bash
# run command to change trace level to DEBUG

A script is provided to collect and package log files of Synthetic PoP into a tar.gz file and send it to Instana support team for assistance. You can find the script in the Instana Helm charts directory or download it from the synthetic-pop-charts repository. Run the following command:

## collect PoP logs from a namespace
./ -n <namespace>