Installing an operator-based Instana setup

1. Pre-requisites

2. Installing the Operator

The Instana operator must be installed with the help of our kubectl plugin. Before we can do so, we must configure a set of Secrets.

2.1 Secret for Docker Registry access

A Secret named instana-registry must be added for accessing the Docker registry. Username is _, password is your download key/agent key.

kubectl create secret docker-registry instana-registry --namespace instana-operator \
    --docker-username=_ \
    --docker-password=<agentKey/downloadKey> \
    --docker-server=containers.instana.io

2.2 TLS secrets for Admission Webhook

The operator comes with an admission webhook for defaulting, validation, and version conversion. TLS is required for an admission webhook, i.e. TLS certificates need to be in place.

The operator expects a secret instana-operator-webhook-certs of type kubernetes.io/tls to be present.

2.2.1 Using cert-manager

By default, we rely on cert-manager to automatically provision the secret. This will work out-of-the-box if cert-manager is installed on the cluster. In this case, Issuer and Certificate objects are created upon installation, which will cause cert-manager to provision and inject the certificates.

Please refer to the cert-manager documentation for installation details:

https://cert-manager.io

2.2.2 Using custom certificates

We do recommend you use cert-manager. However, if you prefer to provision the certificates yourself, you may do so. The secret must contain the following entries:

The certificate (tls.crt) must contain the following DNS names:

Replace the placeholders <namespace> and <clusterDomain> with values that apply for your setup.

$ kubectl create secret instana-operator-webhook-certs \
    --type=kubernetes.io/tls \
    --from-file=tls.key=path/to/tls.key \
    --from-file=tls.crt=path/to/tls.crt \
    --from-file=ca.crt=path/to/ca.crt

$ kubectl create secret instana-operator-webhook-certs \
    --type=kubernetes.io/tls \
    --from-file=tls.key=path/to/tls.key \
    --from-file=tls.crt=path/to/tls.crt \
    --from-file=ca.crt=path/to/ca.crt \
    --dry-run=client \
    --output=yaml

We can now proceed and install the operator. The plugin provides two options:

2.3 Applying Manifests directly

This option creates CRDs and installs the operator deployment and associated resources on the Kubernetes cluster. Please note that special care has to be taken if you use cert-manager in combination with a custom cluster domain (i.e. not cluster.local) or if you generated certificates yourself.

$ kubectl instana operator apply

$ kubectl instana operator apply --namespace=instana-operator

$ kubectl instana operator apply --ca-bundle-base64=<base64-encoded ca.crt file>

2.4 Rendering Manifests only

This options renders manifests to stdout or writes them to files in a given directory.

$ kubectl instana operator template --output-dir <dir>

$ kubectl instana operator template --cluster-domain=my-domain.local

$ kubectl instana operator template --ca-bundle-base64=<base64-encoded ca.crt file>

You can now apply the manifests using kubectl.

2.5 Advanced Operator configuration

The full set of configuration options for the operator is available by using a values file. The mechanism is the same as is used by Helm.

$ kubectl instana operator template --output-dir <dir> --values <path>

Example for a values file:

image:
  registry: my.registry.com

The available options are listed in the following table:

Key Type Default Description
affinity object {} Affinity for the operator pod.
clusterDomain string "" Specifies a custom cluster domain.
extraArgs list [] Additional CLI arguments for the operator process.
extraEnv list [] Additional environment variables for the operator process.
fullnameOverride string "" Overrides the chart's fullname (instana-operator).
image.registry string "containers.instana.io" The image registry to use.
image.repository string "instana/release/selfhosted/operator" The image repository to use.
image.tag string Automatically set by the kubectl plugin The image tag to use.
imagePullPolicy string "IfNotPresent" The image pull policy.
imagePullSecrets list [] A list of image pull secrets.
installCRDs bool true Specifies whether CRDs should be installed.
nameOverride string "" Overrides the chart's name (instana-operator).
nodeSelector object {} Node selector for the operator pod.
podSecurityContext object {"runAsGroup":65532,"runAsNonRoot":true,"runAsUser":65532} Security context for the operator pod.
replicas int 1 The number of replicas to create.
resources object {} Resource requests and limits for the operator pod.
securityContext object {"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]},"readOnlyRootFilesystem":true} Security context for the operator container.
tolerations list [] Tolerations for the operator pod.
webhook.caBundleBase64 string "" Base64-encoded CA bundle for the webhook.

3. Setting up Instana

There is a number of steps that have to be carried out in order to set up Instana. This involves creating a Core object and an associated Unit object. However, before we can do so, we have to create Namespaces and Secrets as shown in the following section.

Tip:

The kubectl plugin has a command to generate YAML templates for namespaces and custom resources in order to help you get started.

kubectl instana template --output-dir <dir>

3.1 Preparation steps

Before we can create Core and Unit, we need Namespaces a set of Secrets for them.

3.1.1 Creating namespaces

Core and Units must be installed in different Namespaces. Each Core needs its own Namespace. Multiple Units that belong to the same Core can be installed in the same Namespace.

Namespace names can be freely chosen. We will use instana-core and instana-units in this guide.

The Instana operator requires a label app.kubernetes.io/name to be present on the namespace. The value must be the name of the namespace. The operator adds these labels if they are missing. It makes sense to add these labels directly, especially when using GitOps for deploying.

apiVersion: v1
kind: Namespace
metadata:
  name: instana-core
  labels:
    app.kubernetes.io/name: instana-core
---
apiVersion: v1
kind: Namespace
metadata:
  name: instana-units
  labels:
    app.kubernetes.io/name: instana-units

Save this to a file, say namespaces.yaml, and apply it.

kubectl apply -f namespaces.yaml

3.1.2 Downloading the license file

Instana requires a license based on your SalesKey for activation. This license file can be optained using the kubectl plugin by running the command

kubectl instana license download --sales-key <SalesKey>

or, alternatively, if you need to manually download the license, you can run the following command:

curl https://instana.io/onprem/license/download/allValid?salesId=<your-SalesKey> -o license.json

3.1.3 Creating secrets

Secret values are not configured via Core and Unit resources. These must go into Kubernetes Secrets. Depending on your configuration, certain Secrets must be in place. Secrets must be created in the Core namespace. They will automatically be propagated to associated Unit namespaces as necessary.

All Secrets must have the following label:

3.1.3.1 Secret instana-registry

Secret of type kubernetes.io/dockerconfigjson for Docker registry access. We've already done this for the operator namespace but will also need it for the Core namespace. Make sure you add required label in this case.

kubectl label secret instana-registry app.kubernetes.io/name=instana
3.1.3.2 Secret instana-base
Key Value
downloadKey The download key you received from us
salesKey The sales key you received from us
token.secret Seed for creating crypto tokens. Pick a random 12 char string
adminPassword The initial password the administrator will receive
license Contents of the license file you downloaded from us
dhparams.pem Diffie-Hellman parameters to use

A Diffie-Hellman parameters file can be generated with the following command:

openssl dhparam -out dhparams.pem 2048

Example using a YAML file

apiVersion: v1
kind: Secret
metadata:
  namespace: instana-core
  name: instana-base
  labels:
    app.kubernetes.io/name: instana
stringData:
  downloadKey: myDownloadKey
  salesKey: mySalesKey
  adminPassword: password
  license: a5941493243a9c119150d21c1c60f5d28123b...
  dhparams.pem: |-
    -----BEGIN DH PARAMETERS-----
    MIIBCAKCAQEA9JkODpu24wyh/+6nfTeUWnSweRsNSlDlxJD0IQZzLZ5xeEsdr2WB
    vAHGyYJUcnupupvrqoJGxXO5yq1fIOEx1IwRR/2pvTvsJk6YmsLP7BERd0UGT8NZ
    dFbdZR0rcO85RovGXrKYIt4pLbs5WgA2SVAS6keVGLqAwbpkmgOOF0zxPFos3as4
    Lx54Nf3JstXxOqX1UbxM1I1aLPWcRaRNHuvRkjnU+LMllXEGmMav+ZRsnYTJ1+mY
    p5n+COnE2T3nJOuF2XdYLO+QcOIedVFwXdSLT1Yp0ELZat6c7Gf259q5hyYHeOoF
    ZCeO7eAVqYP9gu3LP+BOMb1THWvZ6dR/UwIBAg==
    -----END DH PARAMETERS-----
  token.secret: uQOkH+Y4wU_0
type: Opaque

Example using kubectl

kubectl create secret generic instana-base --namespace instana-core \
    --from-literal=downloadKey=myDownloadKey \
    --from-literal=salesKey=mySalesKey \
    --from-literal=adminPassword=password \
    --from-file=license=path/to/license-file \
    --from-file=dhparams.pem=path/to/dhparams.pem \
    --from-literal=token.secret=uQOkH+Y4wU_0
3.1.3.3 Secret instana-service-provider

Contains credentials for SAML/OIDC integration

Key Value
sp.pem The cert/key file
sp.key.pass Password for the cert/key file

An encrypted key for signing/validating messages exchanged with the IDP must be configured. Unencrypted keys won't be accepted.

The following commands can be used to create a combined cert/key file:

openssl genrsa -aes128 -out key.pem 2048

openssl req -new -x509 -key key.pem -out cert.pem -days 365

cat key.pem cert.pem > sp.pem
3.1.3.4 Secret instana-spans

This secret must be present if object storage is used for raw spans storage and credentials are required.

Key Value
accessKeyId Access key ID for the object storage
secretAccessKey Secret access key for the object storage
3.1.3.5 Secret instana-proxy

Required if an HTTP proxy is configured that requires authentication

Key Value
proxyUser Proxy user
proxyPassword Proxy password
3.1.3.6 Secret instana-tls

Required for ingress configuration.

Key Value
tls.crt The TLS certificate for the domain under which Instana is reachable. Must match the in CoreSpec configured baseDomain
tls.key The TLS key

Note that this Secret must be of type kubernetes.io/tls.

Example

kubectl create secret tls instana-tls --namespace instana-core \
    --cert=path/to/tls.cert \
    --key=path/to/tls.key
3.1.3.7 Secret instana-smtp

Required if an SMTP server is used that requires authentication.

Key Value
smtpUser The SMTP user
smtpPassword The password for the SMTP user

3.2. Creating a Core

As we have learned, a Core represents shared components and is responsible for configuring datastore access. As a result, most configuration is going to happen here.

Please see API Reference for details.

A Core custom resource must have version instana.io/v1beta1 and kind Core. Configurations for the Core go into the spec section.

apiVersion: instana.io/v1beta1
kind: Core
metadata:
  namespace: instana-core
  name: instana-core
spec:
  ...

3.2.1 Basic configuration

In this section we will look at basic things to configure.

spec:
  # The domain under whcih Instana is reachable
  baseDomain: instana.example.com

  # This configures an SMTP server for sewnding e-mails.
  # Alternatively, Amazon SES is supported. Please see API reference for details.
  emailConfig:
    smtpConfig:
      from: smtp@example.com
      host: smtp.example.com
      port: 465
      useSSL: true

  # The operator can install network policies for restricting network traffic 
  # to what's required only. By default, network policies are disabled.
  # Set this to true if you want to enable them. We suggest you leave this turned off
  # initially until you've made sure everything works.
  enableNetworkPolicies: true

3.2.2 CPU/Memory resources

The operator uses a set of predefined resource profiles that determine the resources assigned to the individual component pods. The following profiles are available. By default, we use medium if nothing is configured.

spec:
  resoureProfile: large

3.2.3 Agent Acceptor

The acceptor is the endpoint agents need to reach to deliver traces/metrics to Instana. This should usually be a subdomain for the baseDomain configured above.

spec:
  agentAcceptorConfig:
    host: ingress.instana.example.com
    port: 443

3.2.4 Datastores

The following datastores must be configured:

As a minimum, addresses have to be configured for each datastore.

spec:
  datastoreConfigs:
    - type: cassandra
      addresses:
        - 10.164.0.2
    - type: cockroachdb
      addresses:
        - 10.164.0.2
    ...

ClickHouse (default: local) and Elasticsearch (default: onprem_onprem) require a cluster name. If your cluster names don't match the default, you can configure different names:

spec:
  datastoreConfigs:
    - type: clickhouse
      addresses:
        - 10.164.0.2
      clusterName: my_cluster

All databases require a tcp port to be configured. Additionally, clickhouse and elasticsearch also need an http port.

If no ports are configured, the following defaults apply:

Datastore Default Ports
cassandra tcp=9042
cockroachdb tcp=26257
clickhouse tcp=9000, http=8123
elasticsearch tcp=9300, http=9200
kafka tcp=9092
spec:
  datastoreConfigs:
    - type: clickhouse
      addresses:
        - 10.164.0.2
      ports:
        - name: tcp
          port: 8888
        - name: http
          port: 9999
      clusterName: my_cluster

3.3 Creating a Unit

Configuring a Unit is fairly straight forward.

apiVersion: instana.io/v1beta1
kind: Unit
metadata:
  namespace: instana-units
  name: tenant0-unit0
spec:
  # Must refer to the namespace of the associated Core object we created above
  coreName: instana-core

  # Must refer to the name of the associated Core object we created above
  coreNamespace: instana-core

  # The name of the tenant
  tenantName: tenant0

  # The name of the unit within the tenant
  unitName: unit0

  # The agent key that is used for the initial unit setup
  initialAgentKey: myAgentKey

  # The same rules apply as for Cores. May be ommitted. Default is 'medium'
  resourceProfile: large

4. Ingress

You will have to set up load balancers and DNS for the following services:

The following sections describe how this can be done using services of type LoadBalancer. Details depend on your Kubernetes setup.

4.1 Acceptor

apiVersion: v1
kind: Service
metadata:
  namespace: instana-core
  name: loadbalancer-acceptor
spec:
  type: LoadBalancer
  loadBalancerIP: <your loadbalancer IP>
  ports:
    - name: http-service
      port: 443
      protocol: TCP
      targetPort: http-service
  selector:
    app.kubernetes.io/name: instana
    app.kubernetes.io/component: acceptor
    instana.io/group: service

The acceptor DNS name is configured as agentAcceptorConfig in the CoreSpec. Your DNS needs to provide the mapping from this domain to the IP address as an A record.

4.2 Gateway

apiVersion: v1
kind: Service
metadata:
  namespace: instana-core
  name: loadbalancer-gateway
spec:
  type: LoadBalancer
  loadBalancerIP: <your loadbalancer IP>
  ports:
    - name: https
      port: 443
      protocol: TCP
      targetPort: https
    - name: http
      port: 80
      protocol: TCP
      targetPort: http
  selector:
    app.kubernetes.io/name: instana
    app.kubernetes.io/component: gateway
    instana.io/group: service

The gateway DNS name is configured as base_domain in the CoreSpec. You need to set up A records in your DNS for the base_domain and for all tenant unit subdomains.