Using the BeeInstana Kubernetes Operator

Prerequisites

Before you deploy BeeInstana, you need to set up data stores as high-performing and distributed data store cluster. For more information, see Using third-party Kubernetes Operators.

For an offline (air-gapped) installation, complete steps 1, 2, 6 and 7 of the Preparing for an offline (air-gapped) installation section in the Using third-party Kubernetes Operators topic.

Architecture

BeeInstana has four components:

• Kubernetes Operator: The Kubernetes Operator handles deploying, updating, and resizing the BeeInstana instance as configured.
• Configuration service: The configuration service provides the cluster information to the aggregators and the ingestors.
• Ingestor: The ingestor uses the metric data from a Kafka topic, processes it, and sends it to the appropriate aggregator.
• Aggregator: The aggregator writes the metric data to the disk and processes the queries that are made for the stored data.

Sizing BeeInstana on Kubernetes

The amount of resources that is needed for the BeeInstana instance is based on the number of metrics that need to be stored. If this number is not known in advance, then you can calculate by using the following method to get an initial rough estimate:

number_of_metrics = number_of_hosts * 100 * 50

To store 2 million metrics with one aggregator and one ingestor, start with the following resources:

A minimal configuration contains the following resources, which can store 250,000 metrics with one aggregator and one ingestor:

You can modify these resources to suit the environment when the BeeInstana instance is running.

Increase the storage proportional to the duration of retention. For example, if you double the retention period, double the storage.

For the ingestor, it is best to allocate double the amount of average CPU that is used to account for spikes in the incoming metrics. The memory usage is consistent normally, but an extra memory is needed to keep the data in memory when an aggregator is being updated.

Ingestors can be scaled horizontally. You can add or remove the replicas, as needed.

For the aggregator, it is best to allocate double the amount of average CPU that is used to account for user-driven requests. The aggregator makes extensive use of the file system cache and eventually uses all the memory that is provided. Check the total memory resident set size to determine the memory that the aggregator requires. The amount of memory must be double the amount of the resident set size, or else it affects the query performance and the stability.

Aggregator pods are organized into shards. Each aggregator pod within a shard is referred to as a mirror, and stores its own copy of the data in the shard.

Aggregators can be scaled up horizontally by increasing the number of shards. The number of shards can be increased, but never decreased. The aggregator does not support decreasing the number of shards.

The number of mirrors in an aggregator shard affects the availability of BeeInstana. Set the number of mirrors to 2. In this way, the cluster has no loss of data or availability problems when one aggregator mirror is down at a time.

If the number of metrics remains constant, then the aggregator volume size will be steady after 13 months.

Deploying the BeeInstana Kubernetes Operator

The BeeInstana Kubernetes Operator is provided as a Helm chart.

To get the Helm chart and deploy the BeeInstana Kubernetes Operator, complete the following steps:

Updating the BeeInstana Kubernetes Operator

To update the BeeInstana Kubernetes Operator after it is installed as described in the Deploying the BeeInstana Kubernetes Operator section, complete the following steps:

1. Retrieve updates from the Instana Helm repository that is added in step 1 of the Deploying the BeeInstana Kubernetes Operator section by using the following command:

   helm repo update instana
   

2. Perform one of the following steps:

   • For a standard Kubernetes cluster, update the BeeInstana Kubernetes Operator by using the following command:

     helm upgrade beeinstana instana/beeinstana-operator --namespace=beeinstana
     

   • For a Red Hat OpenShift cluster, update the BeeInstana Kubernetes Operator by using the following command:

     helm upgrade beeinstana instana/beeinstana-operator --namespace=beeinstana --set operator.securityContext.seccompProfile.type=RuntimeDefault
     

3. Retrieve the pods in the namespace where BeeInstana is deployed by using the following command:

   kubectl get pod --namespace=beeinstana
   

4. Validate that one BeeInstana Kubernetes Operator pod is present and running. See the following example:

   beeinstana-beeinstana-operator-569999cbfc-5ftf8   1/1     Running   0          1m
   

Updating the BeeInstana Kubernetes Operator does not change the version of BeeInstana that is deployed. The version of BeeInstana that is deployed is configured by setting spec.version as described in the BeeInstana configuration options section.

Deploying the BeeInstana instance on Kubernetes

After you deploy the BeeInstana Kubernetes Operator, you can deploy the BeeInstana instance on Kubernetes. Create a file to store the configuration of BeeInstana. Save this configuration file, and use this file to update the BeeInstana instance in the future.

You can start with the following BeeInstana configuration:

• amd64 and arm64 (offline)

  apiVersion: beeinstana.instana.com/v1beta1
  kind: BeeInstana
  metadata:
    name: instance
    namespace: beeinstana
  spec:
    version: 1.3.13
    imageRegistry: <internal-image-registry>
    adminCredentials:
      secretName: beeinstana-admin-creds
    kafkaSettings:
      brokers:
        # Update KAFKA_NAMESPACE to the namespace where Kafka is installed
        - instana-kafka-bootstrap.<KAFKA_NAMESPACE>:9092
      securityProtocol: SASL_PLAINTEXT
      saslMechanism: SCRAM-SHA-512
      saslPasswordCredential:
        secretName: beeinstana-kafka-creds
    config:
      cpu: 200m
      memory: 200Mi
      replicas: 1
      image:
        name: beeinstana/monconfig
        tag: v2.19.0
    ingestor:
      cpu: 8
      memory: 4Gi
      limitMemory: true
      env: on-prem
      metricsTopic: raw_metrics
      replicas: 1
      image:
        name: beeinstana/ingestor
        tag: v1.85.36-release
    aggregator:
      cpu: 4
      memory: 16Gi
      limitMemory: true
      mirrors: 2
      shards: 1
      volumes:
        live:
          size: 2000Gi
          # Replace <storage_class> with your own storage class in the following line.
          storageClass: <storage_class>
      image:
        name: beeinstana/aggregator
        tag: v1.85.36-release
    useMultiArchImages: true
  

• amd64 and arm64 (online)

  apiVersion: beeinstana.instana.com/v1beta1
  kind: BeeInstana
  metadata:
    name: instance
    namespace: beeinstana
  spec:
    version: 1.3.13
    adminCredentials:
      secretName: beeinstana-admin-creds
    kafkaSettings:
      brokers:
        # Update KAFKA_NAMESPACE to the namespace where Kafka is installed
        - instana-kafka-bootstrap.<KAFKA_NAMESPACE>:9092
      securityProtocol: SASL_PLAINTEXT
      saslMechanism: SCRAM-SHA-512
      saslPasswordCredential:
        secretName: beeinstana-kafka-creds
    config:
      cpu: 200m
      memory: 200Mi
      replicas: 1
    ingestor:
      cpu: 8
      memory: 4Gi
      limitMemory: true
      env: on-prem
      metricsTopic: raw_metrics
      replicas: 1
    aggregator:
      cpu: 4
      memory: 16Gi
      limitMemory: true
      mirrors: 2
      shards: 1
      volumes:
        live:
          size: 2000Gi
          # Replace <storage_class> with your own storage class in the following line.
          storageClass: <storage_class>
    # Should set useMultiArchImages to true for s390x and ppc64le
    useMultiArchImages: false
  

• For IBM Z or LinuxONE:

  apiVersion: beeinstana.instana.com/v1beta1
  kind: BeeInstana
  metadata:
    name: instance
    namespace: beeinstana
  spec:
    version: 1.4.0
    adminCredentials:
      secretName: beeinstana-admin-creds
    kafkaSettings:
      brokers:
        # Update KAFKA_NAMESPACE to the namespace where Kafka is installed
        - instana-kafka-bootstrap.<KAFKA_NAMESPACE>:9092
      securityProtocol: SASL_PLAINTEXT
      saslMechanism: SCRAM-SHA-512
      saslPasswordCredential:
        secretName: beeinstana-kafka-creds
    config:
      cpu: 200m
      memory: 200Mi
      replicas: 1
    ingestor:
      cpu: 8
      memory: 4Gi
      limitMemory: true
      env: on-prem
      metricsTopic: raw_metrics
      replicas: 1
    aggregator:
      cpu: 4
      memory: 16Gi
      limitMemory: true
      mirrors: 2
      shards: 1
      volumes:
        live:
          size: 2000Gi
          # Replace <storage_class> with your own storage class in the following line.
          storageClass: <storage_class>
    # Should set useMultiArchImages to true for s390x and ppc64le
    useMultiArchImages: true
  

• Power (online)

  apiVersion: beeinstana.instana.com/v1beta1
  kind: BeeInstana
  metadata:
    name: instance
    namespace: beeinstana
  spec:
    version: 1.4.0
    kafkaSettings:
      brokers:
        # Update KAFKA_NAMESPACE to the namespace where Kafka is installed
        - instana-kafka-bootstrap.<KAFKA_NAMESPACE>:9092
    config:
      cpu: 200m
      memory: 200Mi
      replicas: 1
    ingestor:
      cpu: 8
      memory: 4Gi
      limitMemory: true
      env: on-prem
      metricsTopic: raw_metrics
      replicas: 1
      image:
        name: beeinstana/ingestor
        registry: artifact-public.instana.io
        tag: v1.164.0-release-linux-ppc64le
    aggregator:
      cpu: 4
      memory: 16Gi
      limitMemory: true
      mirrors: 2
      shards: 1
      image:
        name: beeinstana/aggregator
        registry: artifact-public.instana.io
        tag: v1.164.0-release-linux-ppc64le
      volumes:
        live:
          size: 2000Gi
          # Replace <storage_class> with your own storage class in the following line.
          storageClass: <storage_class>
    # Should set useMultiArchImages to true for s390x and ppc64le
    useMultiArchImages: true
  

Change the configuration as necessary. For more information, see the BeeInstana configuration options section.

Add the following configuration to the BeeInstana spec for the aggregator pods to run:

spec:
  fsGroup: <GROUP_ID>
  seccompProfile:
    type: RuntimeDefault

The file system group ID must be set as described in the Determining the file system group ID on Red Hat OpenShift section.

To easily obtain the group ID, run the following command:

oc get namespace beeinstana -o jsonpath='{.metadata.annotations.openshift\.io\/sa\.scc\.uid-range}' | cut -d/ -f 1

To deploy the BeeInstana instance with the configuration file that was created, complete the following steps:

1. Generate a random password for the BeeInstana admin credentials secret by using the following command:

   PASSWORD=$(base64 < /dev/urandom | head -c32); echo "Password: $PASSWORD";
   

2. Store this password safely in your config.yaml file as follows:

    datastoreConfigs:
      ...
      beeInstanaConfig:
        user: beeinstana-user
        password: <USER_GENERATED_PASSWORD>
      ...
   

3. Create a secret that contains the username and password for connecting to Kafka by running the following command:

   kubectl create secret generic beeinstana-kafka-creds -n beeinstana \
   --from-literal=username=strimzi-kafka-user \
   --from-literal=password=<RETRIEVED_PASSWORD>
   

   Replace <RETRIEVED_PASSWORD> with the password that you obtained when Kafka was deployed on your cluster. For more information, see Creating a SASL/SCRAM user in Kafka.

4. Create a secret that contains the username and password for connecting to BeeInstana by running the following command:

   kubectl create secret generic beeinstana-admin-creds -n beeinstana \
   --from-literal=username=beeinstana-user \
   --from-literal=password=<USER_GENERATED_PASSWORD>
   

5. Apply the configuration to the same namespace where the BeeInstana Kubernetes Operator is deployed. Replace </path/to/configuration/file.yaml> with the path to the configuration file that you created earlier.

   kubectl apply -f </path/to/configuration/file.yaml> --namespace beeinstana
   

6. Wait for the BeeInstana Kubernetes Operator to deploy BeeInstana. To validate the installation, check the pod status.

   kubectl get pods --namespace beeinstana
   

   Validate that pods for all of the components are running. See the following example:

   NAME                                              READY   STATUS    RESTARTS   AGE
   aggregator-0-0                                    1/1     Running   0          2m
   aggregator-0-1                                    1/1     Running   0          75s
   beeinstana-beeinstana-operator-79c8fd74c8-vchh6   1/1     Running   0          169m
   config-546999994-rfvnc                            1/1     Running   0          31m
   ingestor-9c569cbf9-ncrzs                          1/1     Running   0          39s
   

7. Check the deployment. Save the command output in a YAML format and verify the configuration in the YAML file.

   kubectl get beeinstana instance --namespace beeinstana -o yaml
   

   If the deployment status is successful, the status.reconciledAt property is a recent timestamp within the past minute, the status.version property is the configured version, and the status.canBeReconciled property is true. See the following example:

   ...
   status:
     ...
     canBeReconciled: true
     reconciledAt: "2023-03-30T22:47:08Z"
     version: 1.1.5
   

Then, you need to configure Instana to use the deployed BeeInstana instance. For more information, see the Run and configure BeeInstana via Operator topic.

To change an existing BeeInstana instance, modify the configuration file that was saved in the beginning of this section, and repeat steps 2 through 4.

BeeInstana configuration options

You can customize the values of the following keys in the configuration file that is used for deploying the BeeInstana instance on Kubernetes:

Many of the following keys are related to sizing. For more information about how to determine an appropriate value for a key, see the Sizing section.

Determining the file system group ID on Red Hat OpenShift

Red Hat OpenShift requires that file system groups are within a range of values specific to the namespace. On the cluster where the BeeInstana Kubernetes Operator was deployed, run the following command:

kubectl get namespace beeinstana -o yaml

An output similar to the following example is shown for the command:

apiVersion: v1
kind: Namespace
metadata:
  annotations:
    ...
    openshift.io/sa.scc.supplemental-groups: 1000000000/10000
  name: beeinstana

The openshift.io/sa.scc.supplemental-groups annotation contains the range of allowed IDs. The range 1000000000/10000 indicates 10,000 values starting with ID 1000000000, so it specifies the range of IDs from 1000000000 to 1000009999. In this example, the value 1000000000 might be used as a file system group ID.

Removing the BeeInstana instance and the BeeInstana Kubernetes Operator

To remove the BeeInstana instance from Kubernetes, complete the following steps:

1. Delete the Beeinstana instance by running the following command. Replace </path/to/configuration/file.yaml> with the path to the configuration file that was created in deploying the BeeInstana instance on Kubernetes.

   kubectl delete -f </path/to/configuration/file.yaml> --namespace beeinstana
   

2. To confirm the successful removal of the BeeInstana instance, complete the following steps:

   1. Retrieve the Kubernetes pods by running the following command:

      kubectl get pods --namespace beeinstana
      

   2. Confirm that only the BeeInstana Operator pod remains as shown in the following example:

      NAME                                              READY   STATUS    RESTARTS   AGE
      beeinstana-beeinstana-operator-79c8fd74c8-vchh6   1/1     Running   0          169m
      

3. Remove the persistent volume claims by completing the following steps:

   1. Retrieve the persistent volume claims by running the following command:

      kubectl get pvc --namespace beeinstana
      

      The following example shows the retrieved persistent volume claims:

      NAME                                       STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
      live-volumeclaim-template-aggregator-0-0   Bound    pvc-156e2acd-0676-4d6d-9631-0621dbb2c3e9   10Gi       RWO            standard       3m44s
      live-volumeclaim-template-aggregator-0-1   Bound    pvc-beb9ebb2-2137-4222-8643-e7878f5fb2fc   10Gi       RWO            standard       2m35s
      

   2. Delete each aggregator persistent volume claim by running the following command. Replace <NAME> with the persistent volume claim name.

      kubectl delete pvc --namespace beeinstana <NAME>
      

4. Remove the generated cluster configmap by running the following command:

   kubectl delete configmap cluster-42.json --namespace beeinstana
   

5. Remove the created secrets by running the following command:

   kubectl delete secret --namespace beeinstana beeinstana-kafka-creds beeinstana-admin-creds
   

To remove the BeeInstana Operator from Kubernetes, complete the following steps:

1. Uninstall the BeeInstana Kubernetes Operator Helm chart by running the following command:

   helm uninstall beeinstana --namespace beeinstana
   

2. To confirm the successful removal of the Operator, complete the following steps:

   1. Retrieve the Kubernetes pods by running the following command:

      kubectl get pods --namespace beeinstana
      

   2. Confirm that no pods are listed.

To remove the namespace where the BeeInstana Kubernetes Operator and instance were deployed, run the following command:

kubectl delete namespace beeinstana