Using third-party Kubernetes Operators

Each Instana data store is installed as high-performing, distributed data store cluster, and is installed on the same Kubernetes cluster as the Instana backend. The installation uses third-party vendor operators for installing the data stores. Each data store is managed by its respective operator.

You can set up the required data stores by using Kubernetes Operators in your Kubernetes or Red Hat OpenShift cluster by using the following instructions:

Note: To get compatible data store versions for your Instana deployment, install the kubectl plug-in and run the kubectl instana --version command. For more information, see the Instana kubectl plug-in topic.

Preparing for online installation

  1. Install Helm by running the following commands:

    curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh

    Note: The suggested installation and configuration steps for required database operators for Instana are documented in this topic. You might need to make some updates to the installation based on the field requirements.

  2. Ensure that cert-manager, which is used to automatically provision the secret by default, is installed in the cluster where you install the data stores. To install cert-manager, run the command as follows:

    kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.7.1/cert-manager.yaml

  3. Ensure that the clusters for the data store installation have a default StorageClass set.

    • The StorageClasses like premium-rwo are examples that are available with the Kubernetes cluster provider. These StorageClasses must be updated to use RWX or RWO that is available in the cluster where the operators are deployed.

    • Verify the StorageClass by running the following command:

      oc get storageclass -o=jsonpath='{.items[?(@.metadata.annotations.storageclass\.kubernetes\.io\/is-default-class=="true")].metadata.name}'

      If the command doesn't return a value, then you need to set a default StorageClass by running the following command:

      oc patch storageclass <storageclass_name> -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

Preparing for an offline (air-gapped) installation

Mirroring operator images in the image registry (bastion host)

You need to have a bastion host, which can access both the internet and your own internal image registry. The following images are for the pinned Helm chart versions of this document.

# Zookeeper
docker.io/pravega/zookeeper-operator:0.2.14
docker.io/pravega/zookeeper:0.2.13
docker.io/lachlanevenson/k8s-kubectl:v1.23.2
# Kafka
quay.io/strimzi/operator:0.30.0
quay.io/strimzi/kafka:0.30.0-kafka-3.2.0
quay.io/strimzi/kafka-bridge:0.21.6
quay.io/strimzi/jmxtrans:0.30.0
quay.io/strimzi/kaniko-executor:0.30.0
quay.io/strimzi/maven-builder:0.30.0
# Elasticsearch
docker.elastic.co/eck/eck-operator:2.5.0
docker.elastic.co/elasticsearch/elasticsearch:7.16.3
# Postgres
registry.opensource.zalan.do/acid/postgres-operator:v1.9.0
ghcr.io/zalando/spilo-15:2.1-p9
# Cassandra
docker.io/k8ssandra/cass-operator:v1.14.0
docker.io/k8ssandra/cass-management-api:4.0.3
docker.io/k8ssandra/system-logger:v1.14.0
docker.io/datastax/cass-config-builder:1.0.4-ubi7
# Clickhouse
docker.io/altinity/clickhouse-operator:0.19.1
docker.io/altinity/metrics-exporter:0.19.1
registry.access.redhat.com/ubi8/ubi-minimal:8.7-923.1669829893
artifact-public.instana.io/self-hosted-images/k8s/clickhouse:22.3.15.33-1-lts-ibm_v0.16.0

Installing data store Operators

Installing third-party operators offline (air-gapped)

To do an offline (air-gapped) installation for third-party operators, you must pull and host the related operator code to the system where the operator is installed.

  1. Pull required Operators as follows:

    helm pull [chart URL | repo/chartname]
example:
helm pull k8ssandra/cass-operator

  2. Use the gzip file that is downloaded in the previous step to install the respective Operator as follows.

     helm install cass-operator ./cass-operator-0.40.0.tgz -n instana-cassandra --create-namespace --version=0.40.0 --set securityContext runAsGroup=999  --set securityContext.runAsUser=99

Creating a Zookeeper data store by using the Zookeeper Operator

Installing the Zookeeper Operator online

  1. Install the Zookeeper Operator by running the following commands:

    helm repo add pravega https://charts.pravega.io
helm repo update
helm install instana -n instana-zookeeper --create-namespace pravega/zookeeper-operator --version=0.2.14

  2. Create a YAML file as follows:

      apiVersion: "zookeeper.pravega.io/v1beta1"
  kind: "ZookeeperCluster"
  metadata:
    name: "instana-zookeeper"
  spec:
    # For all params and defaults, see https://github.com/pravega/zookeeper-operator/tree/master/charts/zookeeper#configuration
    replicas: 3
    config:
      tickTime: 2000
      initLimit: 10
      syncLimit: 5
      maxClientCnxns: 0
      autoPurgeSnapRetainCount: 20
      autoPurgePurgeInterval: 1
    persistence:
      reclaimPolicy: Delete
      spec:
        resources:
          requests:
            storage: "10Gi"

  3. Apply the deployment by running the following commands:

    kubectl create namespace instana-zookeeper
kubectl apply -f <yaml_file> -n instana-zookeeper

    Note: Replace <yaml_file> with the name of YAML file that you created in the last step.

Verifying Instana ClickHouse

Verify whether Instana ZooKeeper is working by running the following command:

kubectl get all -n instana-zookeeper

If Instana ZooKeeper is working, then the result of the command is as follows:

NAME                                       READY   STATUS    RESTARTS        AGE
pod/chi-instana-local-0-0-0                2/2     Running   0               28d
pod/chi-instana-local-0-1-0                2/2     Running   0               28d
pod/clickhouse-operator-58768cf654-6kq86   2/2     Running   0               28d
pod/instana-zookeeper-0                    1/1     Running   0               28d
pod/instana-zookeeper-1                    1/1     Running   0               28d
pod/instana-zookeeper-2                    1/1     Running   0               28d

NAME                                     TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                                        AGE
service/chi-instana-local-0-0            ClusterIP      None             <none>        8123/TCP,9000/TCP,9009/TCP                     28d
service/chi-instana-local-0-1            ClusterIP      None             <none>        8123/TCP,9000/TCP,9009/TCP                     28d
service/clickhouse-instana               LoadBalancer   x.x.x.x          <pending>     8123:32609/TCP,9000:30633/TCP                  28d
service/clickhouse-operator-metrics      ClusterIP      x.x.x.x.          <none>        8888/TCP                                       28d
service/instana-zookeeper-admin-server   ClusterIP      x.x.x.x.          <none>        8080/TCP                                       28d
service/instana-zookeeper-client         ClusterIP      x.x.x.x.          <none>        2181/TCP                                       28d
service/instana-zookeeper-headless       ClusterIP      None               <none>        2181/TCP,2888/TCP,3888/TCP,7000/TCP,8080/TCP   28d

NAME                                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/clickhouse-operator   1/1     1            1           28d

NAME                                             DESIRED   CURRENT   READY   AGE
replicaset.apps/clickhouse-operator-58768cf654   1         1         1       28d

NAME                                     READY   AGE
statefulset.apps/chi-instana-local-0-0   1/1     28d
statefulset.apps/chi-instana-local-0-1   1/1     28d
statefulset.apps/instana-zookeeper       3/3     28d

Creating a Kafka data store by using Strimzi Operator

Installing the Strimzi Operator

  1. Install the Strimzi Operator by running the following commands:

    helm repo add strimzi https://strimzi.io/charts/
helm repo update
helm install strimzi strimzi/strimzi-kafka-operator --version 0.34.0 -n instana-kafka --create-namespace

  2. Create a YAML file as follows:

    apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: instana
  labels:
    strimzi.io/cluster: instana
spec:
  kafka:
    version: 3.3.2
    replicas: 3
    listeners:
      - name: scram
        port: 9092
        type: internal
        tls: false
        authentication:
          type: scram-sha-512
        configuration:
          useServiceDnsDomain: true
    authorization:
      type: simple
      superUsers:
        - test-kafka-user
    storage:
      type: jbod
      volumes:
        - id: 0
          type: persistent-claim
          size: 50Gi
          deleteClaim: true
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 3
  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      size: 5Gi
      deleteClaim: true
  entityOperator:
    template:
      pod:
        tmpDirSizeLimit: 100Mi
    topicOperator: {}
    userOperator: {}

  3. Apply the deployment by running the following commands:

    kubectl apply -f <yaml_file> -n instana-kafka
kubectl wait kafka/instana --for=condition=Ready --timeout=300s -n instana-kafka

    Note: Replace <yaml_file> with the name of YAML file that you created in the last step.

Verifying the Instana Kafka installation

  1. Verify whether Instana Kafka is installed by running the following command:

    kubectl get all -n instana-kafka

    The example might be as follows:

     NAME                                            READY   STATUS    RESTARTS       AGE
 pod/instana-entity-operator-698b4d79d-57smn     3/3     Running   0              8d
 pod/instana-kafka-0                             1/1     Running   0              8d
 pod/instana-kafka-1                             1/1     Running   0              8d
 pod/instana-kafka-2                             1/1     Running   0              8d
 pod/instana-zookeeper-0                         1/1     Running   0              8d
 pod/instana-zookeeper-1                         1/1     Running   0              8d
 pod/instana-zookeeper-2                         1/1     Running   0              8d
 pod/strimzi-cluster-operator-795f85cf5f-j9jqn   1/1     Running   1 (2d7h ago)   8d

 NAME                               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
 service/instana-kafka-bootstrap    ClusterIP   172.30.89.171   <none>        9091/TCP,9092/TCP            8d
 service/instana-kafka-brokers      ClusterIP   None            <none>        9090/TCP,9091/TCP,9092/TCP   8d
 service/instana-zookeeper-client   ClusterIP   172.30.88.71    <none>        2181/TCP                     8d
 service/instana-zookeeper-nodes    ClusterIP   None            <none>        2181/TCP,2888/TCP,3888/TCP   8d

 NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
 deployment.apps/instana-entity-operator    1/1     1            1           8d
 deployment.apps/strimzi-cluster-operator   1/1     1            1           8d

 NAME                                                  DESIRED   CURRENT   READY   AGE
 replicaset.apps/instana-entity-operator-698b4d79d     1         1         1       8d
 replicaset.apps/strimzi-cluster-operator-795f85cf5f   1         1         1       8d

(Optional) Balancing cluster with Cruise Control

Deploying Cruise Control

First, install the metrics reporters in your cluster’s brokers, and then create a Cruise Control server deployment. To create a Cruise Control server deployment, add the following data to the custom resource of your Kafka:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: instana
spec:
  # ...
  cruiseControl: {}

After you prepare the custom resource of your Kafka configuration file, apply it by running the following command:

kubectl apply -f kafka.yaml -n instana-kafka

When you apply the Kafka custom resource configuration file, all the default Cruise Control settings are used to deploy the Cruise Control server. If you add Cruise Control to an existing cluster (instead of starting a fresh cluster), then you see that all your Kafka broker pods roll to install the metrics reporter.

After the metrics reporter is installed, check whether the Cruise Control is deployed in your cluster by running the following command:

kubectl get pods -n instana-kafka
NAME                                          READY   STATUS    RESTARTS   AGE
instana-cruise-control-6fcbd54b48-ndzrh    2/2     Running   0          8m48s
instana-entity-operator-78d7995695-8tt2v   3/3     Running   0          12m
instana-kafka-0                            2/2     Running   0          9m12s
instana-kafka-1                            2/2     Running   0          10m
instana-kafka-2                            2/2     Running   0          9m46s
instana-zookeeper-0                        1/1     Running   0          13m
instana-zookeeper-1                        1/1     Running   0          13m
instana-zookeeper-2                        1/1     Running   0          13m
strimzi-cluster-operator-54565f8c56-rmdb8     1/1     Running   0          14m

After the Cruise Control is deployed, it starts to pull metrics from the topics that are created by the metrics reporters in each broker.

Getting an optimization proposal

After Cruise Control is deployed, you can generate an optimization proposal. To do this, you need to apply a KafkaRebalance custom resource (the definition for this is installed when you install Strimzi). A basic KafkaRebalance is as follows:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaRebalance
metadata:
  name: my-rebalance
  labels:
    strimzi.io/cluster: instana
# no goals specified, using the default goals from the Cruise Control configuration
spec: {}

You can deploy the KafkaRebalance custom resource like any other resource:

kubectl apply -f kafka-rebalance.yaml -n instana-kafka

When you apply the KafkaRebalance resource to your Kafka cluster, the Cluster Operator issues the relevant requests to Cruise Control to fetch the optimization proposal. The Cluster Operator stores the status of your KafkaRebalance and the progress updates from Cruise Control. You can check by using the following command:

kubectl describe kafkarebalance my-rebalance -n instana-kafka

The result might be as follows:

Name:         my-rebalance
Namespace:    instana-kafka
Labels:       strimzi.io/cluster=instana
Annotations:  API Version:  kafka.strimzi.io/v1beta2
Kind:         KafkaRebalance
Metadata:
# ...
Status:
  Conditions:
    Last Transition Time:  2023-04-04T11:13:35.749066535Z
    Status:                ProposalReady
    Type:                  State
  Observed Generation:     1
  Optimization Result:
    Data To Move MB:  0
    Excluded Brokers For Leadership:
    Excluded Brokers For Replica Move:
    Excluded Topics:
    Intra Broker Data To Move MB:         12
    Monitored Partitions Percentage:      100
    Num Intra Broker Replica Movements:   0
    Num Leader Movements:                 24
    Num Replica Movements:                55
		On Demand Balancedness Score After:   84.23162054159118
    On Demand Balancedness Score Before:  80.87946050436929
    Recent Windows:                       5
  Session Id:                             42f7cc24-b295-4406-80c9-c3c7a3625180

The key parts of the KafkaRebalance request are as follows:

  • The Status.Conditions.Status field shows the progress of your rebalance request. If the proposal is not ready yet, then you might see ProposalPending as the Status.

  • The Status.Optimization Result (once the proposal is ready) shows a summary of the optimization proposal. The meaning of each of these values is described in the Strimzi documentation.

Starting a cluster rebalance

If the KafkaRebalance proposal matches your requirement, then you can apply an approve annotation to the KafkaRebalance resource for Cruise Control to start applying the changes:

kubectl annotate kafkarebalance my-rebalance strimzi.io/rebalance=approve -n instana-kafka

When you apply an approve annotation, Cruise Control updates the changes from the KafkaRebalance proposal. The Cluster Operator keeps the status of the KafkaRebalance resource up-to-date with the progress of the rebalance while it is ongoing. You can check the progress of the rebalance by running the following command:

kubectl describe kafkarebalance my-rebalance -n instana-kafka

When the rebalance is in progress, the Status shows Rebalancing. When the rebalance is finished, the Status shows Ready.

See the following example:

Name:         my-rebalance
Namespace:    instana-kafka
Labels:       strimzi.io/cluster=instana
Annotations:  API Version:  kafka.strimzi.io/v1beta2
Kind:         KafkaRebalance
Metadata:
# ...
Status:
  Conditions:
    Last Transition Time:  2023-04-04T11:16:06.230479824Z
    Status:                Rebalancing
    Type:                  State
# ...
Stopping a cluster rebalance

Rebalances can take a long time and might impact the performance of your cluster. Therefore, you might want to stop a rebalance if that causes an issue. To stop an ongoing rebalance, you can apply the stop annotation to the KafkaRebalance resource at any time by using the following command:

kubectl annotate kafkarebalance my-rebalance strimzi.io/rebalance=stop -n instana-kafka

Creating an Elasticsearch data store by using Elasticsearch (ECK) Operator

Installing the Elasticsearch (ECK) Operator online

  1. Install the Elasticsearch (ECK) Operator by running the following commands:

    helm repo add elastic https://helm.elastic.co
helm repo update
helm install elastic-operator elastic/eck-operator -n instana-elastic --create-namespace --version=2.5.0

  2. Create a YAML file as follows:

    apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: instana
spec:
  version: 7.16.3
  nodeSets:
    - name: default
      count: 3
      config:
        node.master: true
        node.data: true
        node.ingest: true
        node.store.allow_mmap: false
      volumeClaimTemplates:
        - metadata:
            name: elasticsearch-data # Do not change this name unless you set up a volume mount for the data path.
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi
  http:
    tls:
      selfSignedCertificate:
        disabled: true

  3. Apply the deployment by running the following command:

    kubectl apply -f <yaml_file> -n instana-elastic

    Note: Replace <yaml_file> with the name of YAML file that you created in the last step.

Verifying the Elasticsearch Operator deployment

Verify the Elasticsearch Operator deployment by running the following command:

kubectl get all -n instana-elastic

If the Elasticsearch Operator is deployed successfully, then the result of the command is as follows:

NAME                       READY   STATUS    RESTARTS   AGE
pod/elastic-operator-0     1/1     Running   1          4m14s
pod/instana-es-default-1   1/1     Running   0          2m42s
pod/instana-es-default-0   1/1     Running   0          2m42s
pod/instana-es-default-2   1/1     Running   0          2m42s

NAME                               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/elastic-operator-webhook   ClusterIP   10.43.212.82    <none>        443/TCP    4m14s
service/instana-es-transport       ClusterIP   None            <none>        9300/TCP   2m44s
service/instana-es-http            ClusterIP   10.43.204.194   <none>        9200/TCP   2m44s
service/instana-es-default         ClusterIP   None            <none>        9200/TCP   2m42s

NAME                                  READY   AGE
statefulset.apps/elastic-operator     1/1     4m14s
statefulset.apps/instana-es-default   3/3     2m42s

Creating a Postgres data store by using the Zalando Postgres Operator

Installing the Postgres Operator online

  1. Create a SecurityContextConstraints YAML file and a postgresql YAML file as follows:

    apiVersion: security.openshift.io/v1
kind: SecurityContextConstraints
metadata:
  name: postgres-scc
runAsUser:
  type: MustRunAs
  uid: 101
seLinuxContext:
  type: RunAsAny
fsGroup:
  type: RunAsAny
allowHostDirVolumePlugin: false
allowHostNetwork: true
allowHostPorts: true
allowPrivilegedContainer: false
allowHostIPC: true
allowHostPID: true
readOnlyRootFilesystem: false
users:
  - system:serviceaccount:instana-postgres:postgres-operator
  - system:serviceaccount:instana-postgres:postgres-pod
  - system:serviceaccount:instana-postgres:default

    apiVersion: "acid.zalan.do/v1"
kind: postgresql
metadata:
  name: postgres
spec:
  patroni:
    pg_hba:
      - local     all          all                            trust
      - host      all          all          0.0.0.0/0         md5
      - local     replication  standby                        trust
      - hostssl   replication  standby      all               md5
      - hostnossl all          all          all               reject
      - hostssl   all          all          all               md5
  dockerImage: ghcr.io/zalando/spilo-15:2.1-p9
  teamId: instana
  numberOfInstances: 3
  spiloRunAsUser: 101
  spiloFSGroup: 103
  spiloRunAsGroup: 103
  postgresql:
    version: "15"
    parameters:  # Expert section
      shared_buffers: "32MB"
  volume:
    size: 10Gi
    storageClass: premium-rwo    # Optional field. You can assign a non-default StorageClass available in the cluster as needed. If you don't add this field, the default StorageClass is used.
  resources:
    requests:
      cpu: 500m
      memory: 2Gi
    limits:
      cpu: 1000m
      memory: 4Gi

  2. For deploying the Postgres Operator in a Red Hat OpenShift cluster, you need to apply the Security Context Constraints (SCC) before you apply the Postgres Operator deployment:

    kubectl apply -f <yaml_file>

    Note: Replace <yaml_file> with the name of the SecurityContextConstraints YAML file that you created in step 1.

  3. Install the Postgres Operator by running the following commands:

    helm repo add postgres https://opensource.zalando.com/postgres-operator/charts/postgres-operator
helm repo update
helm install postgres-operator postgres/postgres-operator --create-namespace --version=1.9.0 --set configGeneral.kubernetes_use_configmaps=true --set securityContext.runAsUser=101 -n instana-postgres

  4. Apply the Postgres Operator deployment by running the following command:

    kubectl apply -f <yaml_file> --namespace=instana-postgres

    Note: Replace <yaml_file> with the name of the postgresql YAML file that you created in step 1.

Verifying the Postgres Operator deployment

Verify the Postgres Operator deployment by running the following command:

kubectl get all -n instana-postgres

If the Postgres Operator is deployed successfully, then the result of the command is as follows:

NAME                                    READY   STATUS    RESTARTS    AGE
pod/postgres-0                           1/1     Running   0          6d17h
pod/postgres-1                           1/1     Running   0          6d17h
pod/postgres-2                           1/1     Running   0          6d17h
pod/postgres-operator-6695744cb4-b5l6d   1/1     Running   0          34d

NAME                                READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/postgres-operator   1/1     1            1           34d

NAME                                           DESIRED   CURRENT   READY   AGE
replicaset.apps/postgres-operator-6695744cb4   1         1         1       34d

NAME                        READY   AGE
statefulset.apps/postgres   3/3     34d

NAME                                TEAM      VERSION   PODS   VOLUME   CPU-REQUEST   MEMORY-REQUEST   AGE   STATUS
postgresql.acid.zalan.do/postgres   instana   15        3      10Gi     500m          2Gi              34d   Running

NAME                                                    IMAGE                             CLUSTER-LABEL   SERVICE-ACCOUNT   MIN-INSTANCES   AGE
operatorconfiguration.acid.zalan.do/postgres-operator   ghcr.io/zalando/spilo-15:2.1-p9   cluster-name    postgres-pod      -1              34d

Creating a Cassandra data store by using the Cass Operator

Installing the Cass Operator online

  1. Install the Cass Operator for the Cassandra deployment by running the following commands:

    helm repo add k8ssandra https://helm.k8ssandra.io/stable
helm repo update
helm install cass-operator k8ssandra/cass-operator -n instana-cassandra --create-namespace --version=0.40.0 --set securityContext.runAsGroup=999 --set securityContext.runAsUser=999

  2. Create a SecurityContextConstraints YAML file and a CassandraDatacenter YAML file as follows:

    apiVersion: security.openshift.io/v1
kind: SecurityContextConstraints
metadata:
  name: cassandra-scc
runAsUser:
  type: MustRunAs
  uid: 999
seLinuxContext:
  type: RunAsAny
fsGroup:
  type: RunAsAny
allowHostDirVolumePlugin: false
allowHostNetwork: true
allowHostPorts: true
allowPrivilegedContainer: false
allowHostIPC: true
allowHostPID: true
readOnlyRootFilesystem: false
users:
  - system:serviceaccount:instana-cassandra:cass-operator
  - system:serviceaccount:instana-cassandra:default

    apiVersion: cassandra.datastax.com/v1beta1
kind: CassandraDatacenter
metadata:
  name: cassandra
spec:
  clusterName: instana
  serverType: cassandra
  configBuilderImage: docker.io/datastax/cass-config-builder:1.0-ubi7
  serverVersion: "4.0.7"
  managementApiAuth:
    insecure: {}
  size: 3
  allowMultipleNodesPerWorker: false
  resources:
    requests:
      cpu: 2000m
      memory: 8Gi
    limits:
      cpu: 4000m
      memory: 16Gi
  storageConfig:
    cassandraDataVolumeClaimSpec:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 100Gi
  config:
    jvm-server-options:
      initial_heap_size: "4G"
      max_heap_size: "8G"
      additional-jvm-opts:
        - -Dcassandra.allow_unsafe_aggressive_sstable_expiration=true
    cassandra-yaml:
      authenticator: org.apache.cassandra.auth.PasswordAuthenticator
      authorizer: org.apache.cassandra.auth.CassandraAuthorizer
      role_manager: org.apache.cassandra.auth.CassandraRoleManager
      memtable_flush_writers: 8
      auto_snapshot: false
      gc_warn_threshold_in_ms: 10000
      otc_coalescing_strategy: DISABLED
      memtable_allocation_type: offheap_objects
      num_tokens: 256
      enable_drop_compact_storage: true

  3. For deploying the Cass Operator in a Red Hat Openshift cluster, you need to apply the Security Context Constraints (SCC) before you apply the Cass Operator deployment run the following command:

    kubectl apply -f <yaml_file>

    Note: Replace <yaml_file> with the name of the SecurityContextConstraints YAML file that you created in step 2.

  4. Apply the Cass Operator deployment by running the following command:

    kubectl apply -f <yaml_file> -n instana-cassandra

    Note: Replace <yaml_file> with the name of the CassandraDatacenter YAML file that you created in step 2.

Verifying the Cassandra deployment

Verify the deployment of the Cassandra data store by running the following command:

kubectl get pods -n instana-cassandra --selector cassandra.datastax.com/cluster=instana

If the Cassandra data store is deployed successfully, then the result of the command might be as follows:

NAME                            READY   STATUS    RESTARTS   AGE
instana-cassandra-default-sts-1   2/2     Running   0          5m10s
instana-cassandra-default-sts-2   2/2     Running   0          5m10s
instana-cassandra-default-sts-0   2/2     Running   0          5m10s

Creating a Clickhouse data store by using the ClickHouse Operator

Installing the ClickHouse Operator online

  1. Install the ClickHouse Operator by running the following command:

    curl --silent https://raw.githubusercontent.com/Altinity/clickhouse-operator/0.19.1/deploy/operator/clickhouse-operator-install-bundle.yaml | sed 's|kube-system|instana-clickhouse|g' | kubectl apply --filename -

  2. Create the image pull secrets for the ClickHouse image by running the following command:

    kubectl create secret docker-registry clickhouse-image-secret \
  --namespace=instana-clickhouse \
  --docker-username=_ \
  --docker-password=<agent_key> \
  --docker-server=artifact-public.instana.io

  3. Create a YAML file as follows:

    apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: "instana"
spec:
  defaults:
    templates:
      dataVolumeClaimTemplate: instana-clickhouse-data-volume
      logVolumeClaimTemplate: instana-clickhouse-log-volume
  configuration:
    files:
      config.d/storage.xml: |
        <yandex>
          <storage_configuration>
            <disks>
              <default/>
            </disks>
            <policies>
              <logs_policy>
                <volumes>
                  <data>
                    <disk>default</disk>
                  </data>
                </volumes>
              </logs_policy>
            </policies>
          </storage_configuration>
        </yandex>
    clusters:
      - name: local
        templates:
          podTemplate: clickhouse
        layout:
          shardsCount: 1
          replicasCount: 2
    zookeeper:
      nodes:
        - host: instana-zookeeper-headless.instana-clickhouse
    profiles:
      default/max_memory_usage: 10000000000
      default/joined_subquery_requires_alias: 0
      default/max_execution_time: 100
      default/max_query_size: 1048576
      default/use_uncompressed_cache: 0
      default/load_balancing: random
      default/background_pool_size: 32
      default/background_schedule_pool_size: 32
    quotas:
      default/interval/duration: 3600
      default/interval/queries: 0
      default/interval/errors: 0
      default/interval/result_rows: 0
      default/interval/read_rows: 0
      default/interval/execution_time: 0
    users:
      default/password: "clickhouse-pass"
      clickhouse-user/networks/ip: "::/0"
      clickhouse-user/password: "clickhouse-pass"
      # Or
      # Generate password and the corresponding SHA256 hash with:
      # $ PASSWORD=$(base64 < /dev/urandom | head -c8); echo "$PASSWORD"; echo -n "$PASSWORD" | sha256sum | tr -d '-'
      # 6edvj2+d                                                          <- first line is the password
      # a927723f4a42cccc50053e81bab1fcf579d8d8fb54a3ce559d42eb75a9118d65  <- second line is the corresponding SHA256 hash
      # clickhouse-user/password_sha256_hex: "a927723f4a42cccc50053e81bab1fcf579d8d8fb54a3ce559d42eb75a9118d65"
      # Or
      # Generate password and the corresponding SHA1 hash with:
      # $ PASSWORD=$(base64 < /dev/urandom | head -c8); echo "$PASSWORD"; echo -n "$PASSWORD" | sha1sum | tr -d '-' | xxd -r -p | sha1sum | tr -d '-'
      # LJfoOfxl                                  <- first line is the password, put this in the k8s secret
      # 3435258e803cefaab7db2201d04bf50d439f6c7f  <- the corresponding double SHA1 hash, put this below
      # clickhouse-user/password_double_sha1_hex: "3435258e803cefaab7db2201d04bf50d439f6c7f"

  templates:
    podTemplates:
      - name: clickhouse
        spec:
          containers:
            - name: instana-clickhouse
              image: artifact-public.instana.io/self-hosted-images/k8s/clickhouse:22.3.15.33-1-lts-ibm_v0.16.0
              command:
                - clickhouse-server
                - --config-file=/etc/clickhouse-server/config.xml
          imagePullSecrets:
            - name: clickhouse-image-secret
    volumeClaimTemplates:
      - name: instana-clickhouse-data-volume
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 100Gi
      - name: instana-clickhouse-log-volume
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 1Gi

  4. Apply the deployment by running the following command:

    kubectl apply -f instana-clickhouse.yaml -n instana-clickhouse

    Note: Replace <yaml_file> with the name of YAML file that you created in the last step.

Verifying the ClickHouse Operator deployment

Verify the ClickHouse Operator deployment by running the following command:

kubectl get all -n instana-clickhouse

If the ClickHouse Operator is deployed successfully, then the result of the command is as follows:

NAME                                       READY   STATUS    RESTARTS   AGE
pod/clickhouse-operator-6fd75cbd68-lnd2n   2/2     Running   0          170m
pod/chi-instana-local-0-0-0                2/2     Running   0          158m
pod/chi-instana-local-0-1-0                2/2     Running   0          134m
pod/instana-zookeeper-0            1/1     Running   0          187m
pod/instana-zookeeper-1                    1/1     Running   0          189m
pod/instana-zookeeper-2                    1/1     Running   0          181m6

NAME                                  TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
service/clickhouse-operator-metrics      ClusterIP      10.43.114.134   <none>        8888/TCP                        170m
service/clickhouse-instana               LoadBalancer   10.43.184.83    9.46.95.158   8123:30458/TCP,9000:31584/TCP   158m
service/chi-instana-local-0-0            ClusterIP      None            <none>        8123/TCP,9000/TCP,9009/TCP      158m
service/chi-instana-local-0-1            ClusterIP      None            <none>        8123/TCP,9000/TCP,9009/TCP      134m
service/instana-zookeeper-client         ClusterIP   10.43.219.0     <none>        2181/TCP                                       110s
service/instana-zookeeper-headless       ClusterIP   None            <none>        2181/TCP,2888/TCP,3888/TCP,7000/TCP,8080/TCP   110s
service/instana-zookeeper-admin-server   ClusterIP   10.43.206.122   <none>        8080/TCP                                       110s

NAME                                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/clickhouse-operator   1/1     1            1           170m

NAME                                             DESIRED   CURRENT   READY   AGE
replicaset.apps/clickhouse-operator-6fd75cbd68   1         1         1       170m

NAME                                     READY   AGE
statefulset.apps/chi-instana-local-0-0   1/1     158m
statefulset.apps/chi-instana-local-0-1   1/1     134m
statefulset.apps/instana-zookeeper       3/3     189m

Installing BeeInstana

For self-hosted Instana environments on Kubernetes with large metric loads, you are recommended to deploy BeeInstana by using BeeInstana operator as described in Using the BeeInstana Kubernetes Operator.