Deploying self-hosted Synthetic PoP with popctl

You can deploy a self-hosted Synthetic point of presence (PoP) by using the popctl command-line tool.

To deploy a self-hosted Synthetic PoP online, complete the following steps:

  1. Add the Instana repository.

  2. Install the popctl command-line tool.

    If the popctl command-line tool is already installed, upgrade the popctl command-line tool.

  3. Create a cluster.

  4. Install Synthetic PoP.

  5. Install Instana agent.

To deploy a self-hosted Synthetic PoP in an air-gapped environment, complete the following steps:

  1. Add the Instana repository.

  2. Install the popctl command-line tool.

    If the popctl command-line tool is already installed, Upgrade the popctl command-line tool.

  3. Create an air-gapped installation package.

  4. Transfer the air-gapped installation package.

  5. Create a cluster.

  6. Install Synthetic PoP.

  7. Install the Instana agent.

System requirements

popctl supports only the Linux x86_64 operating system.

For information on required resources for Synthetic PoP, see Self-hosted PoP Capacity planning and scaling.

Adding the Instana repository

Add the Instana repository to the sources.list file on your host so that the host can access the installation package.

In an air-gapped environment, perform the procedure on the bastion host.

To add the Instana repository, complete one of the following steps as the root user on your host. In the command, replace <download_key> with your download key.

  • For Red Hat® Enterprise Linux® or CentOS Stream, run the following commands:

    export DOWNLOAD_KEY=<download_key>

cat << EOF > /etc/yum.repos.d/Instana-Product.repo
[instana-product]
name=Instana-Product
baseurl=https://_:$DOWNLOAD_KEY@artifact-public.instana.io/artifactory/rel-rpm-public-virtual/
enabled=1
gpgcheck=0
gpgkey=https://_:$DOWNLOAD_KEY@artifact-public.instana.io/artifactory/api/security/keypair/public/repositories/rel-rpm-public-virtual
repo_gpgcheck=1
EOF

  • For Ubuntu or Debian, run the following commands:

    export DOWNLOAD_KEY=<download_key>

echo 'deb [signed-by=/usr/share/keyrings/instana-archive-keyring.gpg] https://artifact-public.instana.io/artifactory/rel-debian-public-virtual generic main' > /etc/apt/sources.list.d/instana-product.list

cat << EOF > /etc/apt/auth.conf
machine artifact-public.instana.io
   login _
   password $DOWNLOAD_KEY
EOF

wget -nv -O- --user=_ --password="$DOWNLOAD_KEY" https://artifact-public.instana.io/artifactory/api/security/keypair/public/repositories/rel-debian-public-virtual | gpg --dearmor > /usr/share/keyrings/instana-archive-keyring.gpg

  • For SUSE Linux Enterprise Server (SLES), run the following commands:

    export DOWNLOAD_KEY=<download_key>
cat << EOF > /etc/zypp/credentials.d/instana_auth
username=_
password=$DOWNLOAD_KEY
EOF
cat << EOF > /etc/zypp/repos.d/Instana-Product.repo
[instana-product]
name=Instana-Product
baseurl=https://artifact-public.instana.io/artifactory/rel-rpm-public-virtual/?credentials=instana_auth
enabled=1
gpgcheck=0
gpgkey=https://artifact-public.instana.io/artifactory/api/security/keypair/public/repositories/rel-rpm-public-virtual?credentials=instana_auth
repo_gpgcheck=1
EOF

Installing the popctl command-line tool

To install the popctl command-line tool, complete one of the following procedures depending on the operating system.

In an air-gapped environment, perform the procedure on the bastion host.

Installing the popctl command-line tool on Red Hat Enterprise Linux or CentOS Stream

To install the popctl command-line tool on a Red Hat Enterprise Linux or CentOS Stream host, complete the following steps:

  1. Refresh your yum repositories:

    yum clean expire-cache -y

  2. Update the package index files on your host:

    yum update -y

  3. Install popctl:

    yum install -y popctl

  4. If the versionlock plug-in is not installed on your host, install the plug-in:

    yum install python3-dnf-plugin-versionlock

  5. Avoid automatic updates, set the popctl package version:

    yum versionlock add popctl

Installing the popctl command-line tool on Ubuntu or Debian

To install the popctl command-line tool on an Ubuntu or Debian host, complete the following steps:

  1. Update the package index files on your host:

    apt update -y

  2. Install popctl:

    apt install -y popctl

  3. Avoid automatic updates, set the popctl package version:

    apt-mark hold popctl

Installing the popctl command-line tool on SLES

To install the popctl command-line tool on an SLES host, complete the following steps:

  1. Refresh and update the package index files on your host:

    zypper refresh

    When you see the prompt Do you want to reject the key, trust temporarily, or trust always? [r/t/a/?] (r):, enter a to set as trust always.

  2. Install popctl:

    zypper install -y popctl

Upgrading the popctl command-line tool

To upgrade the popctl command-line tool, complete one of the following steps depending on the operating system:

  • For Red Hat Enterprise Linux or CentOS Stream, run the following command:

    yum update popctl

  • For Ubuntu or Debian, run the following command:

    apt update && apt install --only-upgrade popctl

  • For SLES, run the following command:

    zypper refresh && zypper update popctl

After you install the tool, it is automatically added to your $PATH. You can run the popctl commands from any directory.

To check the popctl version:

popctl --version

To check the usage of popctl:

popctl --help

To get help for a specific command:

popctl help [command]

Creating an air-gapped installation package

In an air-gapped environment, you must create an installation package on your bastion host, transfer it to the host that is in the air-gapped environment, then use the package to install. Air-Gapped installation is supported only on single-node K3s cluster. Do not install PoP or Instana agent on other platform such as Red Hat OpenShift, Microk8s, and Minikube.

To create an installation package on your bastion host, complete the following steps:

  1. On your bastion host, make sure that you installed popctl as described in the Installing popctl command-line tool section.

  2. Create the air-gapped package:

    popctl air-gapped package

    By default, the package is created in the current directory.

  3. Enter the download key to continue:

    ? Enter the download key or an official agent key: **********************

    If the key that you entered is valid, the installer starts downloading the artifacts and creates the air-gapped package. You might see the following message:

    ⠋ 1/6 Packaging popctl artifacts [1/1]  [0s] ✓
⠋ 2/6 Packaging BuildMeta artifacts [1/1]  [0s] ✓
⠧ 3/6 Packaging Cluster artifacts [2/2]  [0s] ✓
⠼ 4/6 Packaging Helm artifacts [2/2]  [1m5s] ✓
⠙ 5/6 Packaging Registry (linux/amd64) artifacts [6/6]  [4s] ✓
⠼ 6/6 Packaging Agent (linux/amd64) artifacts [2/2]  [2s] ✓
⠸ Archiving air-gapped package  [5s] ✓

Depending on network conditions, it might take several minutes for the air-gapped package to be created.

When your air-gapped package is ready, you can see the following message:

------------------------------------------
Air-gapped package successfully exported!

File: synthetic-pop-airgapped.tar.gz

Transferring the air-gapped installation package

To transfer the air-gapped package from the bastion host to the host in your air-gapped environment:

  1. Extract the air-gapped package and copy the popctl file to the /usr/local/bin directory on your host:

     tar -xzf </path/to/synthetic-pop-airgapped.tar.gz> -C /usr/local/bin --strip-components 1 airgapped/popctl

  2. Import the air-gapped package:

    popctl air-gapped import -f </path/to/synthetic-pop-airgapped.tar.gz>

Creating a cluster

Create a single-node K3s cluster. If a Kubernetes cluster exists, skip this step.

  • Online:
    popctl cluster create
  • Air-gapped environment:
    popctl cluster create --air-gapped

Installing Synthetic PoP

Install Synthetic PoP:

  • Online:
    popctl up
  • Air-gapped environment:
    popctl up --air-gapped

If a Kubernetes cluster doesn't exist, you can create a cluster and install Synthetic PoP with the following command:

  • Online:
    popctl up --create-k3s-cluster
  • Air-gapped environment:
    popctl up --air-gapped --create-k3s-cluster

To install Synthetic PoP with Redis password, enter the values at the prompt:

? Enter PoP label(label is PoP identifier which can not be changed): synthetic-pop
? Enter instana synthetic endpoint URL: https://synthetics-saas.instana.rocks
? Enter the namespace of Synthetic PoP: default
? Do you want to enable TLS encryption for redis communication? No
? Enter a new password for redis communication: **********
? Enter instana agent key: ************
? Enter the download key: **********************

When installation of the self-hosted PoP is successfully initialized, it starts setting up the cluster. It might take several minutes for the synthetic PoP to be ready. When you see the following message, your Synthetic PoP is ready.

⠹ Setting up the cluster [6s] ✓
⠦ Applying synthetic-pop [1m54s] ✓

To install Synthetic PoP with Redis TLS enabled, enter the parameter values at the prompt:

? Enter PoP label(label is PoP identifier which can not be changed): synthetic-pop-default
? Enter instana synthetic endpoint URL: https://synthetics-saas.instana.rocks
? Do you want to enable TLS encryption for redis communication? Yes
? Enter TLS certificate files path including tls.crt, tls.key and ca.crt (hit Enter to auto-generate):
? Enter instana agent key: ************
? Enter the download key: **********************

When you see the following message, your Synthetic PoP is ready.

⠧ Generate TLS Secrets [5s] ✓
⠋ Setting up TLS Secrets [0s] ✓
⠼ Applying synthetic-pop [1m24s] ✓

Installing Instana agent

Install an Instana agent to monitor Synthetic PoP.

To install the Instana agent, run the following command. You need not pass the --air-gapped parameter.

popctl agent apply

Enter the values at the prompt.

A sample output is shown in the following example:

? popctl will use Helm Chart to install/update Instana Agent, please uninstall the existing Instana Agent first if the agent is installed with Operator or YAML, please confirm to install agent with Helm Chart Yes

? Enter Instana Endpoint port: 443

⠴ Applying instana-agent [5s] ✓
⠋ Setting up TLS Secrets [0s] ✓
⠋ Setting up Instana Agent TLS [0s] ✓

Uninstalling the Instana agent

To unistall the Instana agent, run the popctl agent delete command.

A sample output is shown in the following example:

? Do you want to delete the Instana agent? Yes
⠋ Uninstalling instana-agent from namespace instana-agent [0s] ✓
⠋ Deleting Agent TLS Secrets [0s] ✓

Uninstalling Synthetic PoP

To uninstall Synthetic PoP, run the popctl down command.

A sample output is shown in the following example:

? Do you want to uninstall PoP under namespasce "default"? Yes
⠧ Uninstalling synthetic-pop from namespace default [0s] ✓

Deleting K3s cluster

To delete the K3s cluster, run the popctl cluster delete command.

The K3s binary file is deleted along with the cluster.

Frequently asked questions

Find answers to frequently asked questions that are related to Synthetic PoP.

How to install a new Synthetic PoP in another namespace?

The installation configuration file is saved in ~/.popctl/synthetic-pop.yaml. You can rename it, then rerun popctl up to start a new installation for Synthetic PoP.

How to upgrade the Synthetic PoP or Instana agent?

Rerun the popctl up command and the popctl agent apply to upgrade the Synthetic PoP and Instana agent to the latest version with same configuration.

How to change the parameters for Synthetic PoP or Instana agent?

You can manually update the parameters that are saved in ~/.popctl/synthetic-pop.yaml and rerun the popctl up or popctl agent apply.

How to fix the “Timeout error: context deadline exceeded” issue when you install Synthetic PoP?

Check the status of the pods in the Synthetic PoP namespace by running kubectl get pod -n <namespace>. If a pod is not in ready status, run kubectl logs -n <namespace> -f <pod-name> to check the log messages and see the error message. If a wrong parameter such as a wrong Instana agent key or Synthetic endpoint URL is entered, correct them and run popctl up.

How to fix the "another operation (install/upgrade/rollback) is in progress“ issue when installation is interrupted?

If the popctl up or popctl agent apply command is interrupted unexpectedly, run the same command again.

To list all releases regardless of status, run the helm ls -aA command.

A sample output is shown in the following example:

 NAME         	NAMESPACE    	REVISION	UPDATED                                	STATUS         	CHART               	APP VERSION
 instana-agent	instana-agent	1       	2024-08-21 03:41:38.75709505 -0400 EDT 	pending-install	instana-agent-1.2.73	1.275.0

From the list, you can see that the release status is pending-install. To uninstall the release, run the helm uninstall instana-agent -n instana-agent command. Then, run the popctl up or popctl agent apply command again.

How to pass other parameters such as replica number and trace level when you run popctl up?

You can use --set to pass the parameters defined in the Helm chart values.yaml, similar to the Helm instillation command.

To view the parameters to scale the replicas and update trace level for playback engine pod:

 popctl up \
 --set browserscript.replicas=2 \
 --set ism.replicas=2 \
 --set http.replicas=2 \
 --set javascript.replicas=2

To update trace level:

--set browserscript.traceLevel="DEBUG" \
--set http.traceLevel="DEBUG" \
--set javascript.traceLevel="DEBUG" \
--set ism.traceLevel="DEBUG" \
...