Operational data store

IBM® Automation foundation includes an operational data store based on Apache 2.0 OSS Elasticsearch with the addition of a custom security plug-in to enable Basic Authentication and a proxy sidecar for TLS capability.

Warning: The disks are not monitored for usage and it is the user’s responsibility to plan for the appropriate disk usage and monitor the disks to ensure they do not fill. If the disks get full, it is likely that it results in data loss.

Jump to

• Running in production
• Node groups
• Security
• Storage
• Additional Allowed APIs
• Backup and restore
• Monitoring
• Audit logging
• JVM options
• Elasticsearch on multi-zone clusters
• Support for the latest Elasticsearch version (7.15.1)

Running in production

When you run Elasticsearch in production, consider these settings. For more information, see Important System Configuration .

• Ensure that sufficient virtual memory exists on the nodes. To configure virtual memory, use the node tuning Operator in OpenShift. Set the value of vm.max_map_count to at least 262144. - For more information, see Using the Node Tuning Operator .
• Ensure that the Elasticsearch container JVM is configured with sufficient heap. For more information, see JVM Options.
• For production environments, persistent storage is required to ensure resilient operation. Cluster scaling is only supported with persistence enabled by using compatible block storage and a minimum of three Elasticsearch nodes are recommended for resilience and availability. Single node deployments should be reserved for proof of concept deployments only.

Node groups

Elasticsearch is configured into node groups inside the Elasticsearch CR:

 spec:
   elasticsearch:
     nodegroupspecs:
       - name: master
         replicas: 3
         storage: {}
         config:
           - key: node.master
             value: "true"
           - key: node.data
             value: "false"
           - key: node.ingest
             value: "false"
       - name: data
         replicas: 3
         storage: {}
         config:
           - key: node.master
             value: "false"
           - key: node.data
             value: "true"
           - key: node.ingest
             value: "false"

These node groups allow the user to define the configuration for different sets of nodes within the Elasticsearch cluster.

The names of these node groups are significant as it defines the names of Kubernetes resources, which are created for installing the Elasticsearch cluster. When a node group is removed, all the resources, except for the PersistentVolumeClaims (used for persistent storage under those nodes), are removed. This means that if you change the name of a group, it leads to the removal of the previous group and the creation of a new group. Therefore, you must consider the node groups carefully before installing or adjusting the configuration.

Security

The Elasticsearch cluster included in Automation foundation uses a custom security plug-in that secures all API interactions with Basic Authentication. A CartridgeRequirements Custom Resource (CR) has its status that is updated regarding a secret that contains the username and generated password that is assigned to the CartridgeRequirements request.

A superuser credential is created for administration of the Elasticsearch instance that can access the security APIs.

• A warning message in the status section of the Elasticsearch CR prompts the administrator who creates the Elasticsearch instance to update the generated password.
• To update the password, the administrator needs to update the corresponding secret's password value and remove the annotation.

All communication with the client port of Elasticsearch is encrypted with TLS. The default TLS configuration can be changed by using one of the following methods:

• Configure TLS to use selfsigned generated certificates with spec.elasticsearch.tls: {}.
• Specify a cert-manager Issuer with a reference to the certificate authority (CA)'s public certificate that is used to verify the provided issuer as shown in the following example:

spec:
  elasticsearch:
    tls:
      issuer:
        name: my-issuer
      caSecret:
        secretName: my-ca-secret
        key: ca.crt

You can customize Elasticsearch user credentials. Currently, the behavior is as following.

• If the user wants a particular combination of user name and password to be used for the Elasticsearch instance, then they must create a secret with name <cloudpakName>-es-auth with key-value pairs as follows:

    {"username": "(your-username)", "password": "(your-password)"}
  

• After this when the user applies a CartridgeRequirements CR, the reconcile picks the credentials that are provided in the secret to create the Elasticsearch user account.
• If the user does not provide any secret as stated previously, then the Elasticsearch user credentials are auto-generated, username is the Cloud Pak name and password is an auto generated alphanumeric string.
• If the user wants to update the created credentials, then they can modify the password in the secret <cloudpakName>-es-auth. It takes around 30 secs for the updated password to reflect in the Elasticsearch instance. Note: After the credentials are created, the username must not be changed. Only the password can be modified.

Updating and preconfiguring the superuser password

Follow the process to change the generated password for the elasticsearch-admin superuser:

1. Ensure you are logged in to the OpenShift cluster by using the oc login.

2. Update and export the following environment variables in your command line.

    export ELASTIC_INSTANCE_NAME="elasticsearch-sample" # <-- Change this variable to your CR name.
    export NAMESPACE="acme-abp" # <-- Change this variable to the namespace that you want.
    export NEW_PASSWORD="your-new-password" # <-- Change this variable to the new password that you want.
   

3. Run the following code block in your command line to update the superuser password:

    export SECRET_NAME=$(oc get elasticsearch "$ELASTIC_INSTANCE_NAME" -n "$NAMESPACE" -o jsonpath='{.status.adminAuthSecretName}')
    oc patch secret $SECRET_NAME -n $NAMESPACE -p '{"data": {"password": "'$(echo -n "$NEW_PASSWORD" | base64)'"}}'
    oc annotate secret $SECRET_NAME -n $NAMESPACE elastic.automation.ibm.com/generated-default-credentials-
    unset NEW_PASSWORD
   

Alternatively, to preconfigure the superuser password, a secret can be created with the appropriate naming and labeling conventions in advance. Creating a secret in advance can be useful in a disaster recovery scenario where dependent services have an existing set of credentials.

1. Ensure you are logged in to the OpenShift cluster by using the oc login.

2. Update and export the following environment variables in your command line.

    export ELASTIC_INSTANCE_NAME="elasticsearch-sample" # <-- Change this variable to your CR name.
    export NAMESPACE="acme-abp" # <-- Change this variable to the namespace that you want.
    export NEW_PASSWORD="your-new-password" # <-- Change this variable to the new password that you want.
   

3. Run the following code block in your command line to preconfigure the superuser password:

    cat <<EOF | oc apply -f -
    kind: Secret
    apiVersion: v1
    metadata:
      name: ${ELASTIC_INSTANCE_NAME}-elasticsearch-es-default-user
      namespace: $NAMESPACE
      labels:
        app.kubernetes.io/component: es
        app.kubernetes.io/instance: $ELASTIC_INSTANCE_NAME
        app.kubernetes.io/name: elasticsearch
        elastic.automation.ibm.com/cr-name: $ELASTIC_INSTANCE_NAME
    data:
      password: $(echo -n "$NEW_PASSWORD" | base64 -w0)
      username: ZWxhc3RpY3NlYXJjaC1hZG1pbg==
    type: kubernetes.io/basic-auth
    EOF
    unset NEW_PASSWORD
   

Note: This precreated credentials secret is not owned by the Elasticsearch instance and as such is not tethered to the Elasticsearch instance lifecycle. Users are responsible for managing the lifecycle of this secret.

Storage

By default the Automation foundation provided Elasticsearch cluster does not have persistence that is configured. Cluster administrators are required to provide either a StorageClass supporting dynamic provisioning or pre-created PersistentVolumes before configuring Elasticsearch. There are multiple PersistentVolume storage classes available, depending on your cluster setup. For more information, see Understanding Persistent Storage.

Each node group requires independent storage configuration. This approach enables different tiers of storage capability to be provided to each node group depending on requirements that is, fast storage for the data nodes and slower storage for the controller nodes.

The controller and data nodes require storage that can be used with a ReadWriteOnce (RWO) access mode. This mode specifies that the volume can be mounted as read and write by a single node and is available only to that node.

To use the Elasticsearch cluster data storage, you must declare its use in the elasticsearch section of the AutomationBase custom resource by defining a storage element for each nodegroupspecs.

For example,

spec:
  elasticsearch:
    nodegroupspecs:
      - name: data
        replicas: 3
        storage:
          size: 50Gi
          class: rook-ceph-block

This example creates a 50 GB PersistentVolumeClaim for each of the three replicas where the physical storage for each of the data nodes is provisioned by using the rook-ceph-block StorageClass.

The following table shows the child elements that can be specified as part of the storage object. All elements are optional. If the element is not provided, and a default value is set for the cluster, then that default value is used. Otherwise, the element is not included.

If spec.elasticsearch.nodegroupspecs[].storage.class is not specified, and a default StorageClass is set for the cluster, the default StorageClass is used.

If you don't want a StorageClass to be used because your PersistentVolumes do not define a StorageClass, like NFS, then specify spec.elasticsearch.nodegroupspecs[].storage.class with a value of "", as in class: "".

• If StorageClasses support dynamic provisioning

  • Some StorageClasses support dynamic provisioning. If the StorageClass in use supports dynamic provisioning, then a PersistentVolume is dynamically created when the PersistentVolumeClaim is created. Otherwise, the PersistentVolume needed to be created before the Elasticsearch storage is defined.
  • On OpenShift, all dynamically provisioned volumes are created with the RECLAIMPOLICY set to Delete by default. Thus, the volume lasts only while the claim still exists in the system. If you delete the claim, the volume is also deleted, and all data on the volume is lost.

• If StorageClasses do not support dynamic provisioning

  • If the storage class doesn't support dynamic provisioning, for example, NFS, or if the PersistentVolumes are already created for use with Elasticsearch, the PersistentVolumes need to be referenced from the PersistentVolumeClaim that gets created by the Elasticsearch operator. Use the storage.selector or storage.volumeClaimTemplate child elements of the spec.elasticsearch.nodegroupspecs[] element in the AutomationBase custom resource.

Note: The storage.volumeClaimTemplate child element can be used only when you configure the Elasticsearch storage snapshot.

Create PersistentVolumes by using a label so that they can be correctly defined in the PersistentVolumeClaim, as shown in the following example:

apiVersion: v1
kind: PersistentVolume
metadata:
  name:  elasticsearch-data1
  labels:
    es-storage-type: data
spec:
  capacity:
    storage: 50Gi
  accessModes:
  - ReadWriteOnce
  nfs:
    path: /data/es-data1
    server: 192.168.1.10
  persistentVolumeReclaimPolicy: Recycle

See the following example of the spec.elasticsearch.nodegroupspecs[].storage that uses a selector that matches the es-storage-type: data label:

spec:
  elasticsearch:
    license:
      accept: true
    version: "v1.0"
    nodegroupspecs:
      - name: data
        replicas: 3
        storage:
          class: ""
          selector:
            matchLabels:
              es-storage-type: data
          size: 50Gi

Sufficient PersistentVolumes must be available to meet the needs of the replicas that are defined for each node group. In the previous example, three replicas are defined for the data node group. Thus, at least three PersistentVolumes of sufficient capacity must be available to meet the claim needs.

Note: If separate data and master nodes are being used with persistent storage, then a spec.elasticsearch.nodegroupspecs[].storage object is required for both nodegroups. It is not possible to define persistent storage just for the data nodes. However, different tiers of storage might be assigned to each node group.

If no storage is defined, the Elasticsearch cluster uses only the storage that is local to the container. All indices and configurations are lost when an Elasticsearch container restarts.

When an Elasticsearch cluster is deleted the PersistentVolumeClaims and bound PersistentVolumes remain intact to preserve the data.

Note: If a new Elasticsearch cluster that uses persisted storage is created with the same name as a previous cluster, then any previous PersistentVolumeClaims and hence PersistentVolumes are reused. If the storage object that is defined on the new cluster is changed, for example the StorageClass differs, then the PersistentVolumeClaims and PersistentVolumes need to be manually deleted before the creation of the new Elasticsearch cluster of the same name in order for new PersistentVolumeClaims to be created.

Access modes

The Elasticsearch cluster requires storage that uses the following access modes:

• The master and data nodes require storage that uses a ReadWriteOnce (RWO) access mode that specifies that the volume can be mounted as read/write by a single node.

• Snapshot storage requires a ReadWriteMany (RWX) access mode that specifies that the volume can be mounted as read/write by many nodes.

Only PersistentVolume mechanisms that support these access modes can be used for the different storage uses.

For more information, see Access modes.

Storage permissions

To access provided storage, workloads require permissions. These permissions are controlled with the securityContext for the pod, in this case, the fsGroup, and supplementalGroups. When the securityContext has an fsGroup that is specified, all processes of the containers within the pod are part of the supplementary group ID that is specified by the fsGroup. The owner for the persisted volume and any files that are created in that volume are also in the group ID that is specified by the fsGroup.

The default settings are valid for most scenarios. However, if your storage configuration requires fsGroup or supplementalGroups, use the Storage object to configure each node group.

• If you configure the storage.fsGroup element on the nodegroupspec, this sets the spec.securityContext.fsGroup element on the pods.
• If you configure the storage.supplementalGroups element on the nodegroupspec, this sets the spec.securityContext.supplementalGroups element on the pods.

Note: The spec.securityContext.runAsGroup cannot be set as part of the storage object, so the default primary group ID for all containers is 0 (root).

The following example demonstrates setting an fsGroup and supplementalGroups in the elasticsearch field of the AutomationBase custom resource for a node group:

spec:
  elasticsearch:
    nodegroupspecs:
      - name: data
        replicas: 3
        storage:
          class: rook-ceph-block
          size: 50Gi
          fsGroup: 2000
          supplementalGroups: [2001,2002]

The three pods that result from the previous node group definition each have a securityContext as shown:

spec:
  securityContext:
    runAsNonRoot: true
    fsGroup: 2000
    supplementalGroups: [2001,2002]

If you run the id command within the pod's container, you can confirm if the configurations took effect:

$ id
uid=1000660000(1000660000) gid=0(root) groups=0(root),2000,2001,2002

Note: If fsGroup or supplementalGroups are required, you might need to provide a SecurityContextConstraint that is configured to support the specified values.

See the following example of the settings that are required in a SecurityContextConstraint to accommodate these configurations:

fsGroup:
  ranges:
    - max: 3000
      min: 2000
  type: MustRunAs
supplementalGroups:
  type: RunAsAny

Additional Allowed APIs

Use the additional allowed APIs (additionalAllowedAPIs) to allow a predefined set of APIs and optionally a broader set of user-specified APIs. Configure this list with the spec.elasticsearch.additionalAllowedAPIs field in the AutomationBase CR.

Note: More APIs are allowed at your own risk!

Follow this format: GET:[api, api_two],POST:[api_three]...

• Add the method in capital letters followed by a colon (:).
• Then, surrounded by brackets, [], add the list of API names.
• Separate the list items with commas.

For an example, see the default allowlist:

"GET:[main_action, cat_health_action, nodes_stats_action, get_snapshots_action, snapshot_status_action, recovery_action, cat_recovery_action, get_indices_action, search_action],HEAD:[main_action, get_indices_action],POST:[rest_handler_security_user_add, put_repository_action, restore_snapshot_action, create_snapshot_action, document_create_action_auto_id, document_index_action, document_create_action, search_action, bulk_action],DELETE:[rest_handler_security_user_delete, delete_repository_action, delete_index_action],PUT:[rest_handler_security_user_update, cluster_update_settings_action, put_repository_action, create_index_action, document_index_action, document_create_action]";

To open all APIs to connect OSS Kibana or other third-party applications, use the following wildcard approach:

"GET:[*],PUT:[*],POST:[*],HEAD:[*],DELETE:[*]"

See the following table for the default allowed APIs and their corresponding Elasticsearch documentation names:

Note: APIs that are deprecated are blocked by the allowlist and cannot be enabled.

Reference table for available APIs to add to the allowlist

Backup and restore

Take a snapshot of a running Elasticsearch cluster to back it up.

Snapshot storage requires a ReadWriteMany (RWX) access mode that specifies that the volume can be mounted as read/write by many nodes.

To use the Elasticsearch cluster snapshot storage, declare its use in the elasticsearch section of the AutomationBase custom resource by defining a snapshotStores element as shown in the following example. This example creates a 10 GB volume that is mounted at /usr/share/elasticsearch/snapshots/main by using the csi-cephfs StorageClass:

spec:
  elasticsearch:
    snapshotStores:
      - name: main
        storage:
          class: "csi-cephfs"
          size: "10Gi"

See the following example of the spec.elasticsearch.snapshotStores.storage that uses a volumeClaimTemplate:

spec:
  elasticsearch:
    license:
      accept: true
    version: "v1.0"
    snapshotStores:
      - name: main
        storage:
          volumeClaimTemplate:
            spec:
              accessModes:
                - ReadWriteMany
              volumeMode: Filesystem
              resources:
                requests:
                  storage: 50Gi
              storageClassName: "myStorageClass"
              selector:
                matchLabels:
                  es-storage-type: data

Note: Only the spec section of PersistentVolumeClaim provided for the volumeClaimTemplate field is used when creating the PersistentVolumeClaim resource. Any metadata entries provided under the volumeClaimTemplate is not used.

• The definition of a snapshot store allocates storage for a snapshot repository that is mounted at a path by using the provided name in the following form: /usr/share/elasticsearch/snapshots/<name>.
• Define one or more snapshot stores within the snapshotStores array. This definition does not create the Elasticsearch snapshot repositories but only the volumes for it.

When you use a shared file system to store snapshots, add the file system path or parent directory to the path.repo setting in the elasticsearch.yml file for each main and data node. Define these values in the Elasticsearch custom resource by defining a config element as shown in the following example:

spec:
  elasticsearch:
    snapshotStores:
      - name: main
        storage:
          class: "csi-cephfs"
          size: "10Gi"
    nodegroupspecs:
      - name: data
        replicas: 3
        config:
        - key: path.repo
          value: "/usr/share/elasticsearch/snapshots/main"

If you configure more than one snapshot store, enter the file system paths with a single path.repo config element as shown in the following example:

spec:
  elasticsearch:
    nodegroupspecs:
      - name: data
        replicas: 3
        config:
        - key: path.repo
          value: '["/usr/share/elasticsearch/snapshots/main", "/usr/share/elasticsearch/snapshots/temporary"]'

Then, create the snapshot repositories with the Elasticsearch REST API. The REST API references the mount path of the storage.

For more information, see Snapshot and restore.

Specify the following child elements as part of the snapshotStores array:

The child elements that you can specify as part of the storage object are the same elements that you can define in the Storage section.

Monitoring

By default, monitoring capabilities for Elasticsearch are disabled. To enable them, deploy your Elasticsearch custom resource as shown in the following example:

spec:
  elasticsearch:
    monitoring: {}

This feature deploys resources necessary to export the Elasticsearch metrics from the running instance to a Prometheus-friendly format and exposes an endpoint with these metrics. It also deploys a resource to gather these metrics into OpenShift Monitoring.

To enable the necessary monitoring in OpenShift as of version 4.6, follow the instructions in Enabling monitoring for user-defined projects to set up the correct ConfigMap configurations.

After the monitoring is enabled, go to Monitoring>Metrics to view the extracted metrics.

The following metrics for Elasticsearch can be queried.

Audit logging

Enable the custom security plug-in that is provided with the Elasticsearch cluster in IBM Automation foundation to log authorization audit records in Cloud Auditing Data Format (CADF).

The authorization audit logging is disabled by default. To enable it, send a REST request to the Elasticsearch system.

curl -X PUT -u "<USERNAME>:<PASSWORD>" "http://<you_elasticsearch_host>/_cluster/settings" \
-H 'Content-Type: application/json' \
-d'{
  "transient": {
    "logger.com.ibm.elasticsearch.audit": "AUDIT"
  }
}'

To turn off the logging for authorization, send a REST request to the Elasticsearch system.

curl -X PUT -u "<USERNAME>:<PASSWORD>" "http://<you_elasticsearch_host>/_cluster/settings" \
-H 'Content-Type: application/json' \
-d'{
  "transient": {
    "logger.com.ibm.elasticsearch.audit": "OFF"
  }
}'

The audit logs for each container are written to the base logs directory in the <cluster_name>_audit.json file. For example, if the Elasticsearch instance is called elasticsearch-sample, the log file is /usr/share/elasticsearch/storage/logs/elasticsearch-sample-elasticsearch-cluster_audit.json.

See the following example of a CADF authorization audit log message:

{
    "outcome": "success",
    "typeURI": "http://schemas.dmtf.org/cloud/audit/1.0/event",
    "eventType": "activity",
    "eventTime": "2021-01-12T12:33:21.409062Z",
    "action": "authenticate",
    "requestPath": "/customer/_doc/1",
    "id": "elasticsearch:ccb7c058-0c32-443c-a6ee-acab813ad978",
    "severity": "normal",
    "initiator": {
        "id": "elasticsearch:d367b08e-af48-456f-9f3b-4e325c85a098",
        "name": "elasticsearch-admin",
        "typeURI": "service/security/account/user",
        "host": {
            "agent": "PostmanRuntime/7.26.8",
            "address": "/127.0.0.1"
        },
        "credential": {
            "type": "user"
        }
    },
    "target": {
        "id": "elasticsearch:elasticsearch-sample-elasticsearch-es-master-data-0",
        "name": "elasticsearch-sample-elasticsearch-es-master-data-0",
        "typeURI": "service/security/account/user"
    },
    "observer": {
        "name": "ElasticSearchSecurityPlugin",
        "id": "userActivity",
        "typeURI": "service/security/elasticsearch"
    },
    "reason": {
        "reasonCode": 200,
        "reasonType": "OK"
    }
}

JVM options

The JVM that is being used within the Elasticsearch container is set up using the default Elasticsearch JVM settings. By default, Elasticsearch tells the JVM to use a heap with a minimum and maximum size of 1 GB. When you move to production, it is important to configure heap size to ensure that Elasticsearch has enough heap available. The JVM options can be specified by providing an ES_JAVA_OPTS environment variable for each of the containers.

The following example demonstrates the use of AutomationBase custom resource to set the ES_JAVA_OPTS environment variable, which specifies the Java minimum and maximum heap size for the master-data containers.

spec:
  elasticsearch:
    nodegroupspecs:
      - name: master-data
        replicas: 3
        storage: {}
        template:
          pod:
            spec:
              containers:
                - env:
                    - name: ES_JAVA_OPTS
                      value: '-Xms2g -Xmx2g'
                  name: elasticsearch
                  resources:
                    limits:
                      cpu: 1100m
                      memory: 5Gi
                    requests:
                      cpu: 900m
                      memory: 3Gi

Note: When you set the Xms (minimum heap size) and Xmx (maximum heap size) settings, you must set these to be equal to each other. Also, you must set Xmx and Xms to no more than 50% of your physical RAM.

For more information, see Setting the heap size documentation.

Elasticsearch on multi-zone clusters

The Elasticsearch included in IBM Automation Foundation works fine on multi-zone clusters. Although it works fine and provides resilience against whole-zone failures, it may not keep up optimal performance. The performance depends on low latency and high bandwidth connection between data centres hosting the nodes.

For more details, please refer the link here.

The cross cluster replication feature, required to implement the performance optimised multi-zone deployment is not available in the current version of Elasticsearch.

Support for the latest Elasticsearch version

IBM Automation Foundation v1.3.0 with Elasticsearch Operator version 1.3.0 has included a new Operand version for ElasticSearch CRs of 2.0.0.

The use of the 2.0.0 operand version (and the corresponding v2 operand channel) will utilise the latest Elasticsearch (ELv2 v17.15.1). The v2 operand needs to be set on the elasticsearch element in the Automationbase CR. For example:

apiVersion: base.automation.ibm.com/v1beta1
kind: AutomationBase
metadata:
  name: iaf-automationbase-instance
  namespace: acme-iaf
spec:
  elasticsearch: 
    license: 
      accept: true
    version: v1.0
    monitoring: {}
    nodegroupspecs:
      - name: master-data
        replicas: 3  
    tls: {}
    version: v2
  kafka: {}
  license:
    accept: true
  tls: {}
  version: v1

If the v1 channel is used, Elasticsearch v7.8.0 will continue to be used.

Note: Once the v2 operand is used, it is not possible to easily move to v1. This is due to the resources created by Elasticsearch 7.15.1 is not being valid on 7.8.0. While upgrading from v1 to v2 operator channel, it is highly recommended to make a backup of the Elasicsearch database prior to modifying the operand to use v2.