Installing the agent on Kubernetes

You can install the Instana agent in an online or offline (air-gapped) environment in a Kubernetes cluster. See the following list to check the prerequisite conditions, learn about all the available installation methods, and install the agent on Kubernetes.

Prerequisites

Before you install the host agent, ensure that the prerequisites are met.

Note:
To check the RBAC permissions that the Instana agent operator requires and installs, see Required RBAC.

Choosing how you want to install the agent

The host agent can be installed with the Kubernetes package manager Helm or with an operator. Both installation options are available for offline (air-gapped) and online environments:

For offline (air-gapped) environments where the host can communicate with your self-hosted agent repositories:

For all available configuration options for the Helm chart, see Configuration reference. These configuration options also apply to the Operator CRD in the same way.

Installing the agent in an air-gapped environment

You can install the agent in an air-gapped environment by using two methods:

Install by using Helm chart in an air-gapped environment

To install the agent in an air-gapped environment, complete the following steps:

Note:
In the following installation steps, the CentOS Linux distribution is used, but the steps are valid for all the other supported distributions.
  1. Install Docker by running the following commands:

    sudo yum install -y yum-utils
    sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
    sudo yum install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
    sudo systemctl start docker
  2. Install Helm by running the following commands:

    sudo su
    yum install -y epel-release
    yum install -y snapd
    systemctl enable --now snapd.socket
    ln -s /var/lib/snapd/snap /snap
    snap install helm --classic
    PATH="$PATH:/snap/bin/"
    helm version
  3. Install Minikube by completing the following steps:

    Note:
    These steps serve as an example to show the target environment.
    1. Add the Kubernetes repository:

      cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
      [kubernetes]
      name=Kubernetes
      baseurl=https://pkgs.k8s.io/core:/stable:/v1.28/rpm/
      enabled=1
      gpgcheck=1
      gpgkey=https://pkgs.k8s.io/core:/stable:/v1.28/rpm/repodata/repomd.xml.key
      exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
      EOF
    2. Install the kubectl command line tool:

      sudo yum install -y kubectl
    3. Download the Minikube RPM package:

      curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-latest.x86_64.rpm
    4. Install the Minikube RPM package:

      sudo rpm -Uvh minikube-latest.x86_64.rpm
    5. Start a Kubernetes cluster by using Docker:

      minikube start --force --driver=docker
    6. Set up the Docker environment so that it can be used with Minikube:

      eval $(minikube docker-env)
    7. Check the Kubernetes cluster:

      kubectl cluster-info
  4. Pull the necessary Docker images for transfer to an air-gapped environment:

    1. Log in to the Instana Container Registry with your agent key:

      docker login https://containers.instana.io/v2 -u _ -p <agentKey>
    2. Pull the latest version of the agent from the Instana Container Registry:

      docker pull containers.instana.io/instana/release/agent/static:latest
    3. Pull the latest Instana agent operator image:

      docker pull icr.io/instana/instana-agent-operator:latest
    4. Pull the latest version of the Instana Kubernetes sensor:

      docker pull icr.io/instana/k8sensor:latest
    5. Log out of the Docker registry:

      docker logout
    6. Convert the Docker images into .tar files:

      docker images
      docker tag <instanaAgentImageID> instana/agent
      docker save instana/agent > instana-agent.tar
      docker tag <operatorImagerId> instana/instana-agent-operator
      docker save instana/instana-agent-operator > instana-agent-operator.tar
      docker tag <k8SensorID> instana/k8sensor
      docker save instana/k8sensor > instana-k8sensor.tar
    7. Copy files (instana-agent.tar, instana-agent-operator.tar, and instana-k8sensor.tar) to the air-gapped host where you want to install the agent.

    8. Delete current images from the air-gapped host:

      docker rmi -f <instanaAgentImageID> <operatorImagerId> <k8SensorID>
    9. Import images from .tar files (instana-agent.tar, instana-agent-operator.tar, and instana-k8sensor.tar):

    docker load --input instana-agent.tar
    docker load --input instana-agent-operator.tar
    docker load --input instana-k8sensor.tar
  5. Run the Docker registry server and push images:

    docker run -d -p 5000:5000 --restart=always --name registry registry:2
    docker tag instana/agent:latest localhost:5000/instana-agent
    docker push localhost:5000/instana-agent
    docker tag instana/instana-agent-operator:latest localhost:5000/instana-agent-operator
    docker push localhost:5000/instana-agent-operator
    docker tag instana/k8sensor:latest localhost:5000/instana-k8sensor
    docker push localhost:5000/instana-k8sensor
    #Delete all images related to the agent:
    docker rmi -f <instanaAgentImageID> <k8SensorID> <operatorImagerId>
  6. Pull Instana Helm charts from the following repository:

    helm pull instana-agent --repo https://agents.instana.io/helm --untar instana-agent
    Note:
    Check for the latest Instana agent Helm chart file in your current directory.
  7. Deploy the Instana agent into the Kubernetes cluster.

    Note:
    Instana agent Helm charts version 2.0.0 and earlier do not have the instana-agent/crds folder. Therefore, the kubectl apply -f instana-agent/crds step is not required.
    kubectl apply -f instana-agent/crds && helm upgrade --install --create-namespace \
       --namespace instana-agent \
       --set agent.key=<agentKey> \
       --set agent.endpointHost=ingress-red-saas.instana.io \
       --set agent.endpointPort=443 \
       --set cluster.name='mip-back-test' \
       --set zone.name='mip-gke-zone' \
       --set k8s_sensor.deployment.enabled=true \
       --set k8s_sensor.image.name=localhost:5000/instana-k8sensor \
       --set k8s_sensor.image.tag=latest \
       --set k8s_sensor.image.pullPolicy=IfNotPresent  \
       --set agent.image.name=localhost:5000/instana-agent \
       --set agent.image.tag=latest \
       --set agent.image.pullPolicy=IfNotPresent  \
       --set controllerManager.image.name=localhost:5000/instana-agent-operator \
       --set controllerManager.image.tag=latest \
       instana-agent instana-agent/

    Enter <agentKey>, and edit <agent.endpointHost>, <agent.endpointPort>, <k8_senosr.image.name>, and <agent.image.name>. Also, replace instana-agent-1.2.61.tgz with the latest Instana agent Helm chart file in your directory.

  8. Check your pods with the following command:

    kubectl get all -n instana-agent

The agent is installed. For more information about the next steps, see the What to do next section.

Install by using an operator in an air-gapped environment

In a fully air-gapped environment, the cluster nodes cannot pull images directly from icr.io or containers.instana.io. Use a jump host (bastion) with internet access to copy all required images to a private registry that the cluster can access. Then, deploy the operator from the mirrored images.

The steps in this section use a private namespace in IBM Cloud Container Registry (ICR) as the target registry. Any OCI-compliant registry that is reachable from the jump host and target cluster works equally well. Examples include JFrog Artifactory, Red Hat Quay, Harbor, or a self-hosted Distribution registry. If you use a registry other than ICR, skip the ICR-specific steps and substitute your own registry URL and authentication credentials wherever icr.io appears.

The overall flow is as follows:
[internet] ──pull──▶ jump host ──push──▶ <private-registry>/instana-airgap-mirror ◀──pull── K8s cluster
                      (skopeo)

The scripts/mirror-instana-images.sh script handles mirroring all three components in one invocation:

Table 1. Components and source images
Component Source image
Operator icr.io/instana/instana-agent-operator:latest
Agent (static) containers.instana.io/instana/release/agent/static:latest
Kubernetes sensor icr.io/instana/k8sensor:latest
Note:
The script copies the static agent image. Use the dynamic agent image only if you want automatic dynamic agent updates.
Note:
Unlike OpenShift, plain Kubernetes has no oc secrets link command. Associate pull secrets by patching imagePullSecrets directly onto the relevant service accounts using kubectl patch.

Prerequisites for air-gapped operator installation

Install the following tools on the jump host before you proceed:

Table 2. Required tools for jump host
Tool Purpose Install guide
skopeo Copy container images between registries without a Docker daemon dnf install skopeo and brew install skopeo
jq Parse image metadata JSON returned by skopeo inspect dnf install jq and brew install jq
kubectl Kubernetes CLI See Kubernetes documentation
ibmcloud CLI + Container Registry plug-in Create and authenticate with the ICR namespace ibmcloud plugin install container-registry
Note:
The ibmcloud CLI is only required when ICR is used as the target registry. Skip it if you are using a different registry such as Artifactory, Quay, or Harbor.
Verify that the required tools are available:
skopeo --version
jq --version
kubectl version --client

Installation

To install the agent by using the operator in an air-gapped environment, complete the following steps:

  1. Prepare your private registry and authenticate skopeo to it.
    Note:
    If you are using ICR, complete the following steps. If you are using a different registry (Artifactory, Quay, Harbor, or Distribution), create the equivalent target repository, authenticate skopeo with your registry's credentials, and continue with step 2.
    1. Log in to IBM Cloud and create a dedicated namespace for the mirrored images:
      ibmcloud login --sso
      ibmcloud cr region-set global
      ibmcloud cr namespace-add instana-airgap-mirror
    2. Create a dedicated IAM API key for non-interactive registry access:
      ibmcloud iam api-key-create instana-mirror-key -d "skopeo mirror key"
    3. Export the key without echoing it to the terminal, and verify that the login works:
      read -rs ICR_API_KEY
      skopeo login --username iamapikey --password "$ICR_API_KEY" icr.io
      Note:
      If a previous ibmcloud cr login wrote an identitytoken field into ~/.docker/config.json that skopeo cannot overwrite, open the file, delete the icr.io entry, save it, and retry skopeo login.
  2. Obtain an Instana agent download key.

    The static agent image lives on containers.instana.io, which requires an agent download key. Retrieve it from the Instana UI Settings → Agents → Install Agent, or ask your Instana account team. Store it without echoing:
    read -rs INSTANA_AGENT_DOWNLOAD_KEY
  3. Run the mirror script from the jump host to mirror the images to the private registry. The script resolves the current :latest version for each image, copies both the :latest tag and the versioned tag to the destination, and writes a properties file with all resolved coordinates.
    ./scripts/mirror-instana-images.sh \
      --registry      icr.io \
      --namespace     instana-airgap-mirror \
      --dest-creds    "iamapikey:$ICR_API_KEY" \
      --dest-tls-verify true \
      --agent-download-key "$INSTANA_AGENT_DOWNLOAD_KEY"

    The script produces a mirror-instana-images.properties file that contains the exact image coordinates resolved during mirroring. For example:

    operator_version=2.2.14
    operator_destination=icr.io/instana-airgap-mirror/instana-agent-operator
    agent_version=1.320.3
    agent_destination=icr.io/instana-airgap-mirror/agent
    k8sensor_version=1.4.16
    k8sensor_destination=icr.io/instana-airgap-mirror/k8sensor
    Note:
    Store this file alongside your deployment manifests. You will use the image coordinates it contains to configure the InstanaAgent custom resource in a later step.
  4. Confirm that all three repositories are present in the registry and have the expected tags. For ICR, run the following command:
    ibmcloud cr images --restrict instana-airgap-mirror
  5. Create the target namespace and the pull secret:
    kubectl create namespace instana-agent
    
    kubectl create secret docker-registry icr-pull-secret \
      --docker-server=icr.io \
      --docker-username=iamapikey \
      --docker-password="$ICR_API_KEY" \
      -n instana-agent
  6. Download the latest release manifest and replace the upstream operator image reference with the mirrored one:
    curl -fsSL \
      https://github.com/instana/instana-agent-operator/releases/latest/download/instana-agent-operator.yaml \
      -o instana-agent-operator.yaml
    
    sed 's|image: icr.io/instana/instana-agent-operator:|image: icr.io/instana-airgap-mirror/instana-agent-operator:|g' \
      instana-agent-operator.yaml > instana-agent-operator-mirrored.yaml
    Note:
    This step replaces only the operator image line. You will configure the InstanaAgent custom resource that references the agent and k8sensor images separately in a later step.
  7. Apply the patched manifest and wait for the rollout to complete:
    kubectl apply -f instana-agent-operator-mirrored.yaml
    kubectl rollout status deployment/instana-agent-controller-manager \
      -n instana-agent --timeout=120s
  8. Patch the pull secret onto the operator service account. On plain Kubernetes, no oc secrets link equivalent exists, so you must patch imagePullSecrets directly onto the service account. Complete this step immediately after you apply the manifest to prevent the pod from entering ErrImagePull status before the patch is in place.
    kubectl patch serviceaccount instana-agent-operator -n instana-agent \
      -p '{"imagePullSecrets": [{"name": "icr-pull-secret"}]}'
    If the operator pod is already in ErrImagePull, patch the service account and then delete the failing pod so it is rescheduled:
    kubectl delete pod -n instana-agent -l app.kubernetes.io/name=instana-agent-operator
  9. Pre-create the agent service accounts and patch the pull secret onto each one. The operator creates instana-agent and instana-agent-k8sensor when you first apply the InstanaAgent custom resource. If you pre-create them and patch the secret before you apply the custom resource, you avoid an ErrImagePull error on first pod scheduling.
    kubectl create serviceaccount instana-agent -n instana-agent
    kubectl create serviceaccount instana-agent-k8sensor -n instana-agent
    
    kubectl patch serviceaccount instana-agent -n instana-agent \
      -p '{"imagePullSecrets": [{"name": "icr-pull-secret"}]}'
    kubectl patch serviceaccount instana-agent-k8sensor -n instana-agent \
      -p '{"imagePullSecrets": [{"name": "icr-pull-secret"}]}'
    Note:
    If you already applied the custom resource and pods are in ErrImagePull, run the preceding two kubectl patch serviceaccount commands and then delete the failing pods, so they are rescheduled.
  10. Create the InstanaAgent custom resource manifest. Replace the placeholder values with your actual configuration:
    • Replace <INSTANA_AGENT_KEY> and <INSTANA_AGENT_DOWNLOAD_KEY> with the keys from the Instana UI Agents & Collectors → Install Agents → Kubernetes Operator.
    • Substitute the image tags with the versions from mirror-instana-images.properties.
    • Update the proxy fields to match your environment.
    • Omit the proxy* fields if your cluster has direct connectivity to the Instana backend.
    • Set proxyUser and proxyPassword to empty strings if no proxy authentication is required.
    apiVersion: instana.io/v1
    kind: InstanaAgent
    metadata:
      name: instana-agent
      namespace: instana-agent
    spec:
      zone:
        name: <your_zone_name>
      cluster:
        name: <your_cluster_name>
      agent:
        key: <INSTANA_AGENT_KEY>
        downloadKey: <INSTANA_AGENT_DOWNLOAD_KEY>
        endpointHost: <your-ingress-endpoint>
        endpointPort: "443"
        proxyHost: <proxy-host>
        proxyPort: "3128"
        proxyProtocol: http
        proxyUser: ""
        proxyPassword: ""
        image:
          name: icr.io/instana-airgap-mirror/agent
          tag: "<agent_version>"       # value of agent_version in mirror-instana-images.properties
        env: {}
        configuration_yaml: |
          # You can leave this empty, or use this to configure your Instana agent.
          # See https://docs.instana.io/setup_and_manage/host_agent/on/kubernetes/
      k8s_sensor:
        image:
          name: icr.io/instana-airgap-mirror/k8sensor
          tag: "<k8sensor_version>"    # value of k8sensor_version in mirror-instana-images.properties
    Note:
    No pullSecrets field is required in the custom resource because you provided image pull credentials by patching icr-pull-secret onto the service accounts in the previous step.
  11. Apply the custom resource:
    kubectl apply -f instana-agent-cr.yaml
  12. Verify the deployment by listing the pods in the instana-agent namespace:
    kubectl get pods -n instana-agent

    After the deployment completes, you see one instana-agent-controller-manager pod, one instana-agent DaemonSet pod per node, and three instana-agent-k8sensor pods, all in Running status. After the agent establishes the backend connection through the configured proxy, the cluster appears in the Instana UI under the configured zone and cluster name.

The agent is installed. For more information about the next steps, see the What to do next section.

Installing the agent in an online environment

You are recommended to use the following methods to install the Instana agent onto a Kubernetes cluster:

If you are reinstalling the agent, ensure that the agent has been cleanly uninstalled, including deleting all the agent's cluster level objects. For more information, see Uninstalling the Instana agent on Kubernetes.

New versions of the Operator and the Helm chart are released fairly frequently. To keep up with the latest updates for fixes, improvements, and new features, ensure that you are running the current version of either Operator or Helm chart.

To find the current version of the Operator or the Helm chart, see the following topics:

Install by using the operator

You can use the Kubernetes operator that Instana provides to install the agent. Follow the steps:

  1. On the home page of the Instana UI, click Agents & collectors. On the Instana agents tab, select Install agents.

  2. On the agent deployment catalog page, click the tile Kubernetes - Operator.

  3. Deploy the operator as follows. The command installs the latest Operator.

    kubectl apply -f https://github.com/instana/instana-agent-operator/releases/latest/download/instana-agent-operator.yaml

    After you run this command, the operator is up and running in the namespace instana-agent, and waiting for the instana-agent custom resource to be created.

    The latest tag for the Instana agent Operator image in DockerHub and the Red Hat Registry is not supported.

    To get a new version of the Instana Agent Operator, update to the latest Operator YAML from the Operator's GitHub Releases page as mentioned previously.

  4. Enter the cluster name and the agent zone (optionally) that you want the cluster to be part of.

    The cluster name (<your_cluster_name>) is the customized name of the cluster that is monitored by this daemonset.

    The agent zone (<your_zone_name>) is used to customize the zone grouping, which is displayed on the infrastructure map.

  5. Create a custom resource YAML file by copying the YAML template provided in the Instana UI.

    The YAML template is pre-filled with your agent key, host agent endpoint, cluster name, and agent zone.

  6. Edit the custom resource YAML file:

    • The Instana agent custom resource supports the exact same configuration as the Instana Helm chart. For a detailed list of all the configuration parameters and examples, see Instana Helm chart.

      Also see the following topics:

    • If you want to install the agent in a self-hosted environment and the agent key does not have authority to download from the Instana public artifactory, add the download key as downloadKey: <your_download_key>. See the following example:

      agent:
        key: wPYpH7EGK0ucLaO0Nu7BYw
        downloadKey: m007YDoWNload6kE42yukg
        endpointHost: ...
    • If you want to add TLS encryption for the agent endpoint, use an existing secret or use a certificate and private key:

      • Use an existing secret

        You can use an existing secret of the type kubernetes.io/tls for TLS encryption. But you must provide agent.tls.secretName in the custom resource YAML file.

      • Use a certificate and private key

        The certificate and private key must be base64-encoded. To use this variant, add the following parameters to the custom resource YAML file:

        • agent.tls.certificate
        • agent.tls.key
        Note:
        If agent.tls.secretName is set, then agent.tls.certificate and agent.tls.key are ignored.
    • If you want to deploy the static host agent, configure the custom resource YAML file with the static agent image. To list the static host agent image, set agent.image.name to containers.instana.io/instana/release/agent/static. See the following example:

      spec:
         agent:
           image:
             name: containers.instana.io/instana/release/agent/static
    • Optional: You can replace the following values:

      • agent.env: Use this field to specify environment variables for the agent, such as the proxy configuration for the agent. For more possible environment values, see agent configuration. See the following example:

        spec:
            agent:
              env:
               INSTANA_AGENT_TAGS: staging
      • agent.configuration_yaml: Use this field to specify an agent configuration file configuration.yaml. See the following example:

        spec:
            agent:
              configuration_yaml: |
                # Example of configuration yaml template
                # Host
                com.instana.plugin.host:
                  tags:
                    - 'dev'
                    - 'app1'

        For more information, see Configuring host agents by using the agent configuration file.

    • Optional: You can manually edit the instana-agent.customresource.yaml file to add tolerations to the controller-manager pods. For more information about available configuration options, see the Canonical configuration file.
    • Optional: To customize how the agent discovers and monitors applications on a per-process or namespace basis, see Selective monitoring.
  7. Apply the custom resource YAML file:

    kubectl apply -f instana-agent.customresource.yaml

    Where instana-agent.customresource.yaml is the name of your custom resource YAML file.

    The Operator picks up the configuration from the custom resource YAML file and deploys the Instana agent.

The agent is installed. For more information about the next steps, see to the What to do next section.

To update the agent, see Updating an operator installation.

Install by using the Helm chart

You can use the Helm chart to install the agent. The Helm chart adds the Instana agent to all worker nodes that can be used to schedule workload in your cluster by using a DaemonSet.

To install the Instana agent using a Helm chart, you must first install or upgrade your local helm CLI to version 3.

The Instana agent Helm chart supports Red Hat OpenShift 4.x.

To install the agent by using Helm chart, complete the following steps:

  1. On the home page of the Instana UI, click Agents & collectors. On the Instana Agents tab, select Install agents.

  2. On the agent deployment catalog page, click the tile Kubernetes - Helm chart.

  3. Enter the cluster name and (optionally) the agent zone that you want the cluster to be part of.

    The cluster name (<your_cluster_name>) is the customized name of the cluster, which is monitored by this daemonset.

    The agent zone (<your_zone_name>) is used to customize the zone grouping, which is displayed on the infrastructure map.

    The agent deployment code is updated with the values that you provide. All of the other required parameters are pre-populated in the agent deployment code, which looks like the following example:

    helm install instana-agent
    --repo https://agents.instana.io/helm \
    --namespace instana-agent \
    --create-namespace \
    --set agent.key='<your_agent_key>' \
    --set agent.endpointHost='<your_host_agent_endpoint>' \
    --set agent.endpointPort=443 \
    --set cluster.name='<your_cluster_name>' \
    --set zone.name='<your_zone_name>' \
    instana-agent

    Alternatively, save the values.yaml locally, and make sure to set the required fields manually. Then, run the following command:

    helm install instana-agent
    --repo https://agents.instana.io/helm \
    --namespace instana-agent \
    --create-namespace \
    -f values.yaml
    instana-agent
  4. Edit the agent deployment code:

    • To configure the installation, you can specify the values on the command line by using the --set flag or provide a YAML file with your values by using the -f flag. For a detailed list of all the configuration parameters and examples, see Instana Helm chart. Also, see the following topics:

    • If you want to deploy the static host agent, set the flag --set agent.image.name=containers.instana.io/instana/release/agent/static.

    • If you want to set up TLS encryption for the agent endpoint, use an existing secret or use a certificate and private key:

      • Use an existing secret of type kubernetes.io/tls

        You can use an existing secret of the type kubernetes.io/tls for TLS encryption. But you must specify the secretName with --set 'agent.tls.secretName=<YOUR_SECRET_NAME>' during installation. The files from the provided secret is then included in the agent.

      • Use a certificate and private key

        The certificate and private key must be base64-encoded. To use this variant, run helm install with the following extra parameters:

        --set 'agent.tls.certificate=<YOUR_CERTIFICATE_BASE64_ENCODED>'
        --set 'agent.tls.key=<YOUR_PRIVATE_KEY_BASE64_ENCODED>'
        Note:
        If agent.tls.secretName is set, then agent.tls.certificate and agent.tls.key are ignored.
    • If your Kubernetes environment requires different proxies to access the Helm repository and to connect to the Instana backend, you can configure separate proxy settings during installation. For more information about configuration options, see Separate proxies for repository and the Instana backend.

    • Optional: To customize how the agent discovers and monitors applications on a per-process or namespace basis, see Selective monitoring.
  5. Copy and then run the agent deployment code.

The agent is installed. For more information about the next steps, see the What to do next section.

To update the agent, see Updating an operator installation.