Using the BeeInstana Kubernetes Operator

Prerequisites

Before you deploy BeeInstana, you need to set up data stores as high-performing and distributed data store cluster. For more information, see Using third-party Kubernetes Operators.

For an offline (air-gapped) installation, complete steps 1, 2, 6 and 7 of the Preparing for an offline (air-gapped) installation section in the Using third-party Kubernetes Operators topic.

Architecture

BeeInstana has four components:

• Kubernetes Operator: The Kubernetes Operator handles deploying, updating, and resizing the BeeInstana instance as configured.
• Configuration service: The configuration service provides the cluster information to the aggregators and the ingestors.
• Ingestor: The ingestor uses the metric data from a Kafka topic, processes it, and sends it to the appropriate aggregator.
• Aggregator: The aggregator writes the metric data to the disk and processes the queries that are made for the stored data.

Sizing BeeInstana on Kubernetes

The amount of resources that is needed for the BeeInstana instance is based on the number of metrics that need to be stored. If this number is not known in advance, then use the following method as an initial rough estimate:

number_of_metrics = number_of_instana_agents * 60 * 15

Where 60 is an estimated number of entities that each agent monitors, and 15 is an estimate for the number of metrics per entity. These numbers vary greatly depending on the agent and type of entity, so it might not be accurate in many environments.

A minimal configuration contains the following resources, which can store up to 250,000 metrics with one ingestor replica and one aggregator shard:

To support between 250,000 and 750,000 metrics, keep the same resource level per replica and shard, but configure two or three replicas and shards. More replicas provide extra resiliency. For 750,000 metrics, use this configuration:

To support between 750,000 metrics and 9,000,000 metrics, increase the resources in the three replicas and shards up to this level:

Beyond 9,000,000 metrics, increase the number of replicas and shards as needed. The number of ingestor replicas does not need to be the same as the number of aggregator shards. You might need to increase either one more quickly than the other based on the following considerations:

Ingestor replicas can be added or removed as needed. You can increase the number of aggregator shards; you must not decrease it.

After BeeInstana is running, some key indicators show whether BeeInstana is sized correctly for the environment, or if you must resize BeeInstana.

For the ingestor, take the following items into consideration:

• CPU: Add more CPU cores or increase the number of replicas if the average CPU load exceeds 50%. The remaining CPU might be needed to account for spikes in the incoming metric rate.
• Memory: The memory usage is consistent normally around 30% or lower. The remaining memory is needed as a buffer when aggregators are offline while they are being updated.

For the aggregator, take the following items into consideration:

• Memory: The amount of memory must be double the amount of the resident set size (RSS), otherwise BeeInstana query performance and stability decline.
• Emergency flushes: An aggregator starts an emergency flush when available memory is low. If this occurs regularly, aggregators require more memory or more shards. You can monitor emergency flushes on a custom dashboard with the Aggregator.AggregatorHouseKeeping.EmergencyFlush.sum metric. Use RSS to measure aggregator memory usage because other measurements might be inaccurate if the file system cache uses memory.
• File system: If the number of metrics remains constant, then the aggregator volume size will be steady after 13 months, which is the maximum retention time of metrics in BeeInstana.
• CPU: Add more CPU cores if the other resources are at expected levels but CPU usage regularly exceeds 50% when no queries are running. The remaining CPU might be needed for user-driven queries.

High availability: The aggregator can have more than one shard, where multiple shards increase the metric capacity of BeeInstana. However, each shard can also have a mirror, which contains an identical copy of the metric data. This redundancy provides high availability for BeeInstana because all data is still available when a mirror in a shard fails. When the aggregator is updated, one of the mirrors is always active to avoid downtime during an upgrade. Unless the resource cost of aggregator mirrors is unacceptable, set the number of mirrors to 2.

Deploying the BeeInstana Kubernetes Operator

The BeeInstana Kubernetes Operator is provided as a Helm chart.

To get the Helm chart and deploy the BeeInstana Kubernetes Operator, complete the following steps:

Updating the BeeInstana Kubernetes Operator

To update the BeeInstana Kubernetes Operator after it is installed as described in the Deploying the BeeInstana Kubernetes Operator section, complete the following steps:

1. Retrieve updates from the Instana Helm repository that is added in step 1 of the Deploying the BeeInstana Kubernetes Operator section by using the following command:

   helm repo update instana
   

2. Perform one of the following steps:

   • For a standard Kubernetes cluster, update the BeeInstana Kubernetes Operator by using the following command:

     helm upgrade beeinstana instana/beeinstana-operator --namespace=beeinstana
     

   • For a Red Hat OpenShift cluster, update the BeeInstana Kubernetes Operator by using the following command:

     helm upgrade beeinstana instana/beeinstana-operator --namespace=beeinstana --set operator.securityContext.seccompProfile.type=RuntimeDefault
     

3. Retrieve the pods in the namespace where BeeInstana is deployed by using the following command:

   kubectl get pod --namespace=beeinstana
   

4. Validate that one BeeInstana Kubernetes Operator pod is present and running. See the following example:

   beeinstana-beeinstana-operator-569999cbfc-5ftf8   1/1     Running   0          1m
   

Updating the BeeInstana Kubernetes Operator does not change the version of BeeInstana that is deployed. The version of BeeInstana that is deployed is configured by setting spec.version as described in the BeeInstana configuration options section.

Deploying the BeeInstana instance on Kubernetes

After you deploy the BeeInstana Kubernetes Operator, you can deploy the BeeInstana instance on Kubernetes. Create a file to store the configuration of BeeInstana. Save this configuration file, and use this file to update the BeeInstana instance in the future.

You can start with the following BeeInstana configuration:

• amd64 and arm64 (offline)

  apiVersion: beeinstana.instana.com/v1beta1
  kind: BeeInstana
  metadata:
    name: instance
    namespace: beeinstana
  spec:
    version: 1.3.13
    imageRegistry: <internal-image-registry>
    adminCredentials:
      secretName: beeinstana-admin-creds
    kafkaSettings:
      brokers:
        # Update KAFKA_NAMESPACE to the namespace where Kafka is installed
        - instana-kafka-bootstrap.<KAFKA_NAMESPACE>:9092
      securityProtocol: SASL_PLAINTEXT
      saslMechanism: SCRAM-SHA-512
      saslPasswordCredential:
        secretName: beeinstana-kafka-creds
    config:
      cpu: 200m
      memory: 200Mi
      replicas: 1
      image:
        name: beeinstana/monconfig
        tag: v2.19.0
    ingestor:
      cpu: 8
      memory: 4Gi
      limitMemory: true
      env: on-prem
      metricsTopic: raw_metrics
      replicas: 1
      image:
        name: beeinstana/ingestor
        tag: v1.85.36-release
    aggregator:
      cpu: 4
      memory: 16Gi
      limitMemory: true
      mirrors: 2
      shards: 1
      volumes:
        live:
          size: 2000Gi
          # Replace <storage_class> with your own storage class in the following line.
          storageClass: <storage_class>
      image:
        name: beeinstana/aggregator
        tag: v1.85.36-release
    useMultiArchImages: true
  

• amd64 and arm64 (online)

  apiVersion: beeinstana.instana.com/v1beta1
  kind: BeeInstana
  metadata:
    name: instance
    namespace: beeinstana
  spec:
    version: 1.3.13
    adminCredentials:
      secretName: beeinstana-admin-creds
    kafkaSettings:
      brokers:
        # Update KAFKA_NAMESPACE to the namespace where Kafka is installed
        - instana-kafka-bootstrap.<KAFKA_NAMESPACE>:9092
      securityProtocol: SASL_PLAINTEXT
      saslMechanism: SCRAM-SHA-512
      saslPasswordCredential:
        secretName: beeinstana-kafka-creds
    config:
      cpu: 200m
      memory: 200Mi
      replicas: 1
    ingestor:
      cpu: 8
      memory: 4Gi
      limitMemory: true
      env: on-prem
      metricsTopic: raw_metrics
      replicas: 1
    aggregator:
      cpu: 4
      memory: 16Gi
      limitMemory: true
      mirrors: 2
      shards: 1
      volumes:
        live:
          size: 2000Gi
          # Replace <storage_class> with your own storage class in the following line.
          storageClass: <storage_class>
    # Should set useMultiArchImages to true for s390x and ppc64le
    useMultiArchImages: false
  

• For IBM Z or LinuxONE:

  apiVersion: beeinstana.instana.com/v1beta1
  kind: BeeInstana
  metadata:
    name: instance
    namespace: beeinstana
  spec:
    version: 1.4.0
    adminCredentials:
      secretName: beeinstana-admin-creds
    kafkaSettings:
      brokers:
        # Update KAFKA_NAMESPACE to the namespace where Kafka is installed
        - instana-kafka-bootstrap.<KAFKA_NAMESPACE>:9092
      securityProtocol: SASL_PLAINTEXT
      saslMechanism: SCRAM-SHA-512
      saslPasswordCredential:
        secretName: beeinstana-kafka-creds
    config:
      cpu: 200m
      memory: 200Mi
      replicas: 1
    ingestor:
      cpu: 8
      memory: 4Gi
      limitMemory: true
      env: on-prem
      metricsTopic: raw_metrics
      replicas: 1
    aggregator:
      cpu: 4
      memory: 16Gi
      limitMemory: true
      mirrors: 2
      shards: 1
      volumes:
        live:
          size: 2000Gi
          # Replace <storage_class> with your own storage class in the following line.
          storageClass: <storage_class>
    # Should set useMultiArchImages to true for s390x and ppc64le
    useMultiArchImages: true
  

• Power (online)

  apiVersion: beeinstana.instana.com/v1beta1
  kind: BeeInstana
  metadata:
    name: instance
    namespace: beeinstana
  spec:
    version: 1.4.0
    kafkaSettings:
      brokers:
        # Update KAFKA_NAMESPACE to the namespace where Kafka is installed
        - instana-kafka-bootstrap.<KAFKA_NAMESPACE>:9092
    config:
      cpu: 200m
      memory: 200Mi
      replicas: 1
    ingestor:
      cpu: 8
      memory: 4Gi
      limitMemory: true
      env: on-prem
      metricsTopic: raw_metrics
      replicas: 1
      image:
        name: beeinstana/ingestor
        registry: artifact-public.instana.io
        tag: v1.164.0-release-linux-ppc64le
    aggregator:
      cpu: 4
      memory: 16Gi
      limitMemory: true
      mirrors: 2
      shards: 1
      image:
        name: beeinstana/aggregator
        registry: artifact-public.instana.io
        tag: v1.164.0-release-linux-ppc64le
      volumes:
        live:
          size: 2000Gi
          # Replace <storage_class> with your own storage class in the following line.
          storageClass: <storage_class>
    # Should set useMultiArchImages to true for s390x and ppc64le
    useMultiArchImages: true
  

Change the configuration as necessary. For more information, see the BeeInstana configuration options section.

Add the following configuration to the BeeInstana spec for the aggregator pods to run:

spec:
  fsGroup: <GROUP_ID>
  seccompProfile:
    type: RuntimeDefault

The file system group ID must be set as described in the Determining the file system group ID on Red Hat OpenShift section.

To easily obtain the group ID, run the following command:

oc get namespace beeinstana -o jsonpath='{.metadata.annotations.openshift\.io\/sa\.scc\.uid-range}' | cut -d/ -f 1

To deploy the BeeInstana instance with the configuration file that was created, complete the following steps:

1. Generate a random password for the BeeInstana admin credentials secret by using the following command:

   PASSWORD=$(base64 < /dev/urandom | head -c32); echo "Password: $PASSWORD";
   

2. Store this password safely in your config.yaml file as follows:

    datastoreConfigs:
      ...
      beeInstanaConfig:
        user: beeinstana-user
        password: <USER_GENERATED_PASSWORD>
      ...
   

3. Create a secret that contains the username and password for connecting to Kafka by running the following command:

   kubectl create secret generic beeinstana-kafka-creds -n beeinstana \
   --from-literal=username=strimzi-kafka-user \
   --from-literal=password=<RETRIEVED_PASSWORD>
   

   Replace <RETRIEVED_PASSWORD> with the password that you obtained when Kafka was deployed on your cluster. For more information, see Creating a SASL/SCRAM user in Kafka.

4. Create a secret that contains the username and password for connecting to BeeInstana by running the following command:

   kubectl create secret generic beeinstana-admin-creds -n beeinstana \
   --from-literal=username=beeinstana-user \
   --from-literal=password=<USER_GENERATED_PASSWORD>
   

5. Apply the configuration to the same namespace where the BeeInstana Kubernetes Operator is deployed. Replace </path/to/configuration/file.yaml> with the path to the configuration file that you created earlier.

   kubectl apply -f </path/to/configuration/file.yaml> --namespace beeinstana
   

6. Wait for the BeeInstana Kubernetes Operator to deploy BeeInstana. To validate the installation, check the pod status.

   kubectl get pods --namespace beeinstana
   

   Validate that pods for all of the components are running. See the following example:

   NAME                                              READY   STATUS    RESTARTS   AGE
   aggregator-0-0                                    1/1     Running   0          2m
   aggregator-0-1                                    1/1     Running   0          75s
   beeinstana-beeinstana-operator-79c8fd74c8-vchh6   1/1     Running   0          169m
   config-546999994-rfvnc                            1/1     Running   0          31m
   ingestor-9c569cbf9-ncrzs                          1/1     Running   0          39s
   

7. Check the deployment. Save the command output in a YAML format and verify the configuration in the YAML file.

   kubectl get beeinstana instance --namespace beeinstana -o yaml
   

   If the deployment status is successful, the status.reconciledAt property is a recent timestamp within the past minute, the status.version property is the configured version, and the status.canBeReconciled property is true. See the following example:

   ...
   status:
     ...
     canBeReconciled: true
     reconciledAt: "2023-03-30T22:47:08Z"
     version: 1.1.5
   

Then, you need to configure Instana to use the deployed BeeInstana instance. For more information, see the Run and configure BeeInstana via Operator topic.

To change an existing BeeInstana instance, modify the configuration file that was saved in the beginning of this section, and repeat steps 2 through 4.

BeeInstana configuration options

You can customize the values of the following keys in the configuration file that is used for deploying the BeeInstana instance on Kubernetes:

Many of the following keys are related to sizing. For more information about how to determine an appropriate value for a key, see the Sizing section.

You can also customize the retention time for metrics in BeeInstana. Metrics are rolled up into intervals that cover ten seconds, one minute, five minutes, and one hour. Metrics that are rolled up into each of the preceding intervals have a different retention time. To customize the retention time, you can set environment variable overrides in the spec.aggregator section of the configuration file. The rollup intervals and retention times are configured in seconds. The environment variables and default retention times are shown in the following table:

The following example shows how retention values are entered in the configuration file. The values in the example are all default values and do not change the metric retention times, but they can be changed. The rollup interval times cannot be changed.

spec:
...
  aggregator:
    environmentOverrides:
    - name: FLAGS_retention_10s
      value: "86400"
    - name: FLAGS_retention_60s
      value: "2592000"
    - name: FLAGS_retention_300s
      value: "7776000"
    - name: FLAGS_retention_3600s
      value: "34128000"
...

Determining the file system group ID on Red Hat OpenShift

Red Hat OpenShift requires that file system groups are within a range of values specific to the namespace. On the cluster where the BeeInstana Kubernetes Operator was deployed, run the following command:

kubectl get namespace beeinstana -o yaml

An output similar to the following example is shown for the command:

apiVersion: v1
kind: Namespace
metadata:
  annotations:
    ...
    openshift.io/sa.scc.supplemental-groups: 1000000000/10000
  name: beeinstana

The openshift.io/sa.scc.supplemental-groups annotation contains the range of allowed IDs. The range 1000000000/10000 indicates 10,000 values starting with ID 1000000000, so it specifies the range of IDs from 1000000000 to 1000009999. In this example, the value 1000000000 might be used as a file system group ID.

Removing the BeeInstana instance and the BeeInstana Kubernetes Operator

To remove the BeeInstana instance from Kubernetes, complete the following steps:

1. Delete the Beeinstana instance by running the following command. Replace </path/to/configuration/file.yaml> with the path to the configuration file that was created in deploying the BeeInstana instance on Kubernetes.

   kubectl delete -f </path/to/configuration/file.yaml> --namespace beeinstana
   

2. To confirm the successful removal of the BeeInstana instance, complete the following steps:

   1. Retrieve the Kubernetes pods by running the following command:

      kubectl get pods --namespace beeinstana
      

   2. Confirm that only the BeeInstana Operator pod remains as shown in the following example:

      NAME                                              READY   STATUS    RESTARTS   AGE
      beeinstana-beeinstana-operator-79c8fd74c8-vchh6   1/1     Running   0          169m
      

3. Remove the persistent volume claims by completing the following steps:

   1. Retrieve the persistent volume claims by running the following command:

      kubectl get pvc --namespace beeinstana
      

      The following example shows the retrieved persistent volume claims:

      NAME                                       STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
      live-volumeclaim-template-aggregator-0-0   Bound    pvc-156e2acd-0676-4d6d-9631-0621dbb2c3e9   10Gi       RWO            standard       3m44s
      live-volumeclaim-template-aggregator-0-1   Bound    pvc-beb9ebb2-2137-4222-8643-e7878f5fb2fc   10Gi       RWO            standard       2m35s
      

   2. Delete each aggregator persistent volume claim by running the following command. Replace <NAME> with the persistent volume claim name.

      kubectl delete pvc --namespace beeinstana <NAME>
      

4. Remove the generated cluster configmap by running the following command:

   kubectl delete configmap cluster-42.json --namespace beeinstana
   

5. Remove the created secrets by running the following command:

   kubectl delete secret --namespace beeinstana beeinstana-kafka-creds beeinstana-admin-creds
   

To remove the BeeInstana Operator from Kubernetes, complete the following steps:

1. Uninstall the BeeInstana Kubernetes Operator Helm chart by running the following command:

   helm uninstall beeinstana --namespace beeinstana
   

2. To confirm the successful removal of the Operator, complete the following steps:

   1. Retrieve the Kubernetes pods by running the following command:

      kubectl get pods --namespace beeinstana
      

   2. Confirm that no pods are listed.

To remove the namespace where the BeeInstana Kubernetes Operator and instance were deployed, run the following command:

kubectl delete namespace beeinstana