Administrator guide

IBM Automation foundation offers managed instances of Kafka, Elasticsearch (operational data store) for Cloud Paks to use. These instances can be customized by using the AutomationBase custom resource (CR) as described here.

Jump to

Customizing configurations

The AutomationBase custom resource (CR) represents an instance of these services that are managed by IBM Automation foundation that is installed into a Kubernetes namespace.

This custom resource contains the following instance-wide configuration information:

Note: For more information, see AutomationBase CR.

Customizing configurations for Kafka

The AutomationBase custom resource contains the instance-wide configuration information for IBM Automation foundation Kafka instance. You can configure Kafka based on your needs by using the spec.kafka section within the AutomationBase CR.

The default Kafka configuration that is used by IBM Automation foundation can be found here.

For the complete Kafka configuration, see Kafka schema reference External link icon.

Customizing configurations for Apicurio

The AutomationBase custom resource contains the instance-wide configuration information for IBM Automation foundation Apicurio instance. You can configure Apicurio based on your needs by using the spec.apicurio section within the AutomationBase CR. Following is an example of Apicurio configuration:

apiVersion: base.automation.ibm.com/v1beta1
kind: AutomationBase
metadata:
  name: sample
  namespace: iafdemo
spec:
  apicurio:
    resources:
      limits:
        cpu: "1"
        memory: 1300Mi
      requests:
        cpu: 100m
        memory: 600Mi
    config:
      quarkus.log.category."io.apicurio".level: DEBUG
  kafka: {}
  license:
    accept: true
  tls: {}

Customizing configurations for ElasticSearch

ElasticSearch can be enabled with default configuration for the IBM Automation foundation instance by setting the field in the AutomationBase CR to {}. The default ElasticSearch configuration that is used is shown in the following example.

apiVersion: base.automation.ibm.com/v1beta1
kind: AutomationBase
metadata:
  name: sample
  namespace: iafdemo
spec:
  elasticsearch: 
    license:
      accept: true
    nodegroupspecs:
      - name: master-data
        replicas: 3
        storage: {}
        template:
          pod:
            spec: {}
    tls: {}
    version: 1.0.0
  kafka: {}
  license:
    accept: true
  tls: {}
  version: 1.0.0

The spec.elasticsearch.nodegroupspecs[].template.pod.spec set to {} provides a default configuration equivalent to the following.

apiVersion: base.automation.ibm.com/v1beta1
kind: AutomationBase
metadata:
  name: sample
  namespace: iafdemo
spec:
  elasticsearch: 
    license:
      accept: true
    nodegroupspecs:
      - name: master-data
        replicas: 3
        storage: 
          size: 50Gi
        template:
          pod:
            spec:
              affinity:
                nodeAffinity:
                  requiredDuringSchedulingIgnoredDuringExecution:
                    nodeSelectorTerms:
                      - matchExpressions:
                          - key: kubernetes.io/arch
                            operator: In
                            values:
                              - amd64
                podAntiAffinity:
                  preferredDuringSchedulingIgnoredDuringExecution:
                    - weight: 80
                      podAffinityTerm:
                        labelSelector:
                          matchLabels:
                            app.kubernetes.io/instance: iaf-system
                            elastic.automation.ibm.com/node-group-name: master-data
                        topologyKey: kubernetes.io/hostname
              containers:
                - resources:
                    limits:
                      cpu: '5'
                      memory: 10Gi
                    requests:
                      cpu: '3'
                      memory: 5Gi
                  readinessProbe:
                    exec:
                      command:
                        - live.sh
                    initialDelaySeconds: 45
                    timeoutSeconds: 10
                    periodSeconds: 30
                    successThreshold: 1
                    failureThreshold: 5
                  name: elasticsearch
                  livenessProbe:
                    exec:
                      command:
                        - live.sh
                    initialDelaySeconds: 55
                    timeoutSeconds: 10
                    periodSeconds: 40
                    successThreshold: 1
                    failureThreshold: 5
                  imagePullPolicy: IfNotPresent
                  image: >-
                    cp.icr.io/cp/iaf-elasticsearch@sha256:b5a47ce8bfff5328a88da2c1235f403948ca075d90a9ec6bae7cf257a13751a9
                - resources:
                    limits:
                      cpu: '1'
                      memory: 2Gi
                    requests:
                      cpu: 500m
                      memory: 500Mi
                  readinessProbe:
                    httpGet:
                      path: /ready
                      port: 32010
                      scheme: HTTP
                    initialDelaySeconds: 20
                    timeoutSeconds: 10
                    periodSeconds: 30
                    successThreshold: 1
                    failureThreshold: 3
                  name: tls-proxy
                  livenessProbe:
                    httpGet:
                      path: /live
                      port: 32010
                      scheme: HTTP
                    initialDelaySeconds: 30
                    timeoutSeconds: 10
                    periodSeconds: 40
                    successThreshold: 1
                    failureThreshold: 3
                  imagePullPolicy: IfNotPresent
                  image: >-
                    cp.icr.io/cp/iaf-elasticsearch-proxy@sha256:4eb369621cf192851482df44543b812c629aa328a5bd9c9890645b46e10d0e3b
    tls: {}
    version: 1.0.0
  kafka: {}
  license:
    accept: true
  tls: {}
  version: 1.0.0

For more information, see operational data store.

Flink cluster is configured by using the EventProcessor CR. For more information, see EventProcessor CR.

The following example shows an EventProcessor CR definition with Flink configuration options.

apiVersion: eventprocessing.automation.ibm.com/v1beta1
kind: EventProcessor
metadata:
  name: acme-smartdecisions-eventprocessor
  namespace: acme-cp4a
  annotations:
    com.ibm.automation.cartridge: com.acme.smartdecisions-dashboard
spec:
  version: "v1.0"
  license:
    accept: true
  flink:
    taskManager: 
      replicas: 2
      ...
    jobManager: 
      ...
    properties:
      ...
    logConfig: ...
    tls: {}
    authentication: {}
    audit: {}
    storage: {}
    monitoring: {}
  ...
status:
  endpoints:
    - name: FlinkJobManager
      type: API
      scope: Internal
      uri: https://some-service.namespace.svc:8081
      caSecret:
        secretName: my-ca-secret
        key: ca.crt
      authentication:
        type: BasicSecret
        secretName: my-auth-secret

For more information on configuring and working with Flink, see here.

Day 2 operations for Kafka

The AutomationBase custom resource provides the mechanism to manage the Kafka instance created by IBM Automation foundation. The spec.kafka section in the AutomationBase CR is used for configuring Kafka and is compatible with Strimzi custom resource specifications. For more information on day 2 operations on Kafka, see here External link icon.

Day 2 operations for Apicurio

Apicurio uses the underlying Kafka instance to store schema data and metadata. To provide resilience for schema storage, it is suggested to replicate topics storage-topic and global-id-topic by using persistent storage across multiple Kafka brokers.

The Apicurio instance REST API is available on the iaf-system-apicurio OpenShift route. It uses basic authentication through the cartridge Kafka username and SCRAM password that is the user <cartridge-name>-kafka-auth, with the password taken from the secret of the same name. An authenticated user has access to only schemas within the same cartridge namespace that is schemas whose names are prefixed with the cartridge technical name. Following are the sample curl commands to read and write schema resources to the Apicurio REST API.

curl command to write a schema:

curl --cacert <cacert> -u <user>:<pwd> -X POST -H "Content-type: application/json; artifactType=AVRO" -H "X-Registry-ArtifactId: mycartridge_price" --data '{"type":"record","name":"mycartridge_price","namespace":"iaf","fields":[{"name":"symbol","type":"string"},{"name":"price","type":"string"}]}' https://<iaf-system-apicurio-route>/api/artifacts

curl command to read a schema:

curl --cacert <cacert> -u <user>:<pwd> https://<iaf-system-apicurio-route>/api/artifacts/mycartridge_price

Information on using Maven to write schemas can be found in the Apicurio documentation External link icon.

Information on configuring Java clients to handle schemas can be found in the Apicurio documentation External link icon.

Day 2 operations for operational data store

Updating the Elasticsearch superuser password

You can change the generated password for the elasticsearch-admin superuser by using the following steps:

  1. Ensure that you are logged in to the OpenShift cluster by using the oc login.

  2. Update and export the following environment variables in your terminal.

     export ELASTIC_INSTANCE_NAME="elasticsearch-sample" # <-- Change this variable to your CR name.
 export NAMESPACE="acme-abp" # <-- Change this variable to the namespace that you want.
 export NEW_PASSWORD="your-new-password" # <-- Change this variable to the new password that you want.

  3. Run the following commands in your command line to update the superuser password.

     export SECRET_NAME=$(oc get elasticsearch "$ELASTIC_INSTANCE_NAME" -n "$NAMESPACE" -o jsonpath='{.status.adminAuthSecretName}')
 oc patch secret $SECRET_NAME -n $NAMESPACE -p '{"data": {"password": "'$(echo -n "$NEW_PASSWORD" | base64)'"}}'
 oc annotate secret $SECRET_NAME -n $NAMESPACE elastic.automation.ibm.com/generated-default-credentials-
 unset NEW_PASSWORD

Maintaining Elasticsearch users

The Elasticsearch cluster that is included in Automation foundation uses a custom security plug-in that secures all API interactions with basic authentication. The Elasticsearch users are managed by REST APIs, which enable users to be created, updated, and deleted. The superuser credential that is created for administration of the Elasticsearch instance must be used to access the REST API.

Create an Elasticsearch user

  1. Ensure that you are logged in to the OpenShift cluster by using the oc login.

  2. Edit and enter the following commands.

    export ELASTIC_INSTANCE_NAME="elasticsearch-sample" # <-- Change this to your CR name.
export NAMESPACE="acme-abp" # <-- Change this to the wanted namespace.
export CA_CERT_DIR="/tmp" # <-- Optionally change this directory. It's a temporary location for the CA certificate to exist.
export NEW_ES_USERNAME="your-new-es-username"  # <-- Change this variable to the new username
export NEW_ES_PASSWORD="your-new-es-password"  # <-- Change this variable to the new password

  3. Run the following commands.

     export PASSWORD_SECRET_NAME=$(oc get elasticsearch "$ELASTIC_INSTANCE_NAME" -n "$NAMESPACE" -o jsonpath='{.status.endpoints[?(@.scope=="External")].authentication.secret.secretName}')
 export PASSWORD=$(oc get secret "$PASSWORD_SECRET_NAME" -n "$NAMESPACE" -o jsonpath='{.data.password}' | base64 -d)
 export URI=$(oc get elasticsearch "$ELASTIC_INSTANCE_NAME" -n "$NAMESPACE" -o jsonpath='{.status.endpoints[?(@.scope=="External")].uri}')
 if [[ -z $(oc get elasticsearch "$ELASTIC_INSTANCE_NAME" -n "$NAMESPACE" -o jsonpath='{.status.endpoints[?(@.scope=="External")].caSecret}') ]]
 then
   curl -XPOST -H "Content-Type: Application/Json" -u "elasticsearch-admin:${PASSWORD}" "${URI}/_security/users" -d '{"username": "'${NEW_ES_USERNAME}'", "password": "'${NEW_ES_PASSWORD}'"}'
 else
   export CA_SECRET_KEY=$(oc get elasticsearch "$ELASTIC_INSTANCE_NAME" -n "$NAMESPACE" -o jsonpath='{.status.endpoints[?(@.scope=="External")].caSecret.key}')
   export CA_SECRET_NAME=$(oc get elasticsearch "$ELASTIC_INSTANCE_NAME" -n "$NAMESPACE" -o jsonpath='{.status.endpoints[?(@.scope=="External")].caSecret.secretName}')
   oc extract secret/$CA_SECRET_NAME --keys=$CA_SECRET_KEY --to=$CA_CERT_DIR
   curl -XPOST -H "Content-Type: Application/Json" -u "elasticsearch-admin:${PASSWORD}" "${URI}/_security/users" -d '{"username": "'${NEW_ES_USERNAME}'", "password": "'${NEW_ES_PASSWORD}'"}' --cacert "${CA_CERT_DIR}/${CA_SECRET_KEY}"
 fi
 unset NEW_ES_USERNAME
 unset NEW_ES_PASSWORD

Update an Elasticsearch user

  1. Ensure that you are logged in to the OpenShift cluster by using the oc login.

  2. Edit and enter the following commands.

    export ELASTIC_INSTANCE_NAME="elasticsearch-sample" # <-- Change this to your CR name.
export NAMESPACE="acme-abp" # <-- Change this to the wanted namespace.
export CA_CERT_DIR="/tmp" # <-- Optionally change this directory. It's a temporary location for the CA certificate to exist.
export ES_USERNAME="your-es-username"  # <-- Change this variable to the username to update
export NEW_ES_PASSWORD="your-new-es-password"  # <-- Change this variable to the new password

  3. Run the following commands.

     export PASSWORD_SECRET_NAME=$(oc get elasticsearch "$ELASTIC_INSTANCE_NAME" -n "$NAMESPACE" -o jsonpath='{.status.endpoints[?(@.scope=="External")].authentication.secret.secretName}')
 export PASSWORD=$(oc get secret "$PASSWORD_SECRET_NAME" -n "$NAMESPACE" -o jsonpath='{.data.password}' | base64 -d)
 export URI=$(oc get elasticsearch "$ELASTIC_INSTANCE_NAME" -n "$NAMESPACE" -o jsonpath='{.status.endpoints[?(@.scope=="External")].uri}')
 if [[ -z $(oc get elasticsearch "$ELASTIC_INSTANCE_NAME" -n "$NAMESPACE" -o jsonpath='{.status.endpoints[?(@.scope=="External")].caSecret}') ]]
 then
   curl -XPUT -H "Content-Type: Application/Json" -u "elasticsearch-admin:${PASSWORD}" "${URI}/_security/users/${ES_USERNAME}" -d '{"password": "'${NEW_ES_PASSWORD}'"}'
 else
   export CA_SECRET_KEY=$(oc get elasticsearch "$ELASTIC_INSTANCE_NAME" -n "$NAMESPACE" -o jsonpath='{.status.endpoints[?(@.scope=="External")].caSecret.key}')
   export CA_SECRET_NAME=$(oc get elasticsearch "$ELASTIC_INSTANCE_NAME" -n "$NAMESPACE" -o jsonpath='{.status.endpoints[?(@.scope=="External")].caSecret.secretName}')
   oc extract secret/$CA_SECRET_NAME --keys=$CA_SECRET_KEY --to=$CA_CERT_DIR
   curl -XPUT -H "Content-Type: Application/Json" -u "elasticsearch-admin:${PASSWORD}" "${URI}/_security/users/${ES_USERNAME}" -d '{"password": "'${NEW_ES_PASSWORD}'"}' --cacert "${CA_CERT_DIR}/${CA_SECRET_KEY}"
 fi
 unset ES_USERNAME
 unset NEW_ES_PASSWORD

Delete an Elasticsearch user

  1. Ensure you are logged in to the OpenShift cluster by using the oc login.

  2. Edit and enter the following commands.

    export ELASTIC_INSTANCE_NAME="elasticsearch-sample" # <-- Change this to your CR name.
export NAMESPACE="acme-abp" # <-- Change this to the wanted namespace.
export CA_CERT_DIR="/tmp" # <-- Optionally change this directory. It's a temporary location for the CA certificate to exist.
export ES_USERNAME="your-es-username"  # <-- Change this variable to the username to delete

  3. Run the following commands.

     export PASSWORD_SECRET_NAME=$(oc get elasticsearch "$ELASTIC_INSTANCE_NAME" -n "$NAMESPACE" -o jsonpath='{.status.endpoints[?(@.scope=="External")].authentication.secret.secretName}')
 export PASSWORD=$(oc get secret "$PASSWORD_SECRET_NAME" -n "$NAMESPACE" -o jsonpath='{.data.password}' | base64 -d)
 export URI=$(oc get elasticsearch "$ELASTIC_INSTANCE_NAME" -n "$NAMESPACE" -o jsonpath='{.status.endpoints[?(@.scope=="External")].uri}')
 if [[ -z $(oc get elasticsearch "$ELASTIC_INSTANCE_NAME" -n "$NAMESPACE" -o jsonpath='{.status.endpoints[?(@.scope=="External")].caSecret}') ]]
 then
   curl -XDELETE -H "Content-Type: Application/Json" -u "elasticsearch-admin:${PASSWORD}" "${URI}/_security/users/${ES_USERNAME}"
 else
   export CA_SECRET_KEY=$(oc get elasticsearch "$ELASTIC_INSTANCE_NAME" -n "$NAMESPACE" -o jsonpath='{.status.endpoints[?(@.scope=="External")].caSecret.key}')
   export CA_SECRET_NAME=$(oc get elasticsearch "$ELASTIC_INSTANCE_NAME" -n "$NAMESPACE" -o jsonpath='{.status.endpoints[?(@.scope=="External")].caSecret.secretName}')
   oc extract secret/$CA_SECRET_NAME --keys=$CA_SECRET_KEY --to=$CA_CERT_DIR
   curl -XDELETE -u "elasticsearch-admin:${PASSWORD}" "${URI}/_security/users/${ES_USERNAME}" --cacert "${CA_CERT_DIR}/${CA_SECRET_KEY}"
 fi
 unset ES_USERNAME

You can configure IBM Automation foundation to use Flink tools to manage the internal Flink cluster.

Updating authentication credentials

The basic authentication credentials for the JobManager API can be updated after the Flink cluster is up and running.

  1. Find the generated secret named <EP_instance_name>-eventprocessor-ep-admin-user.

  2. Edit the secret and change the values for username or password based on your needs. Now you must also remove the eventprocessing.automation.ibm.com/generated-default-credentials annotation to remove the warning condition on the EventProcessor CR.

Enabling external access

By default, the Flink Job Manager endpoint is not accessible from outside the cluster. You can enable external access by creating your own OpenShift Route.

Note: Enable Flink Job Manager endpoint only in a test environment. Do not enable it in a production environment.

The following snippet is an example of the spec for the route.

kind: Route
apiVersion: route.openshift.io/v1
metadata:
  name: <your_route_name>
  namespace: <your_namespace>
spec:
  to:
    kind: Service
    name: <your_ep_instance_name>-eventprocessor-ep-jobmanager
    weight: 100
  port:
    targetPort: proxy-client
  tls:
    termination: reencrypt
    # key: (a custom key can be supplied for this route endpoint)
    # certificate: (a custom key can be supplied for this route endpoint)
    # ca: (a CA can be supplied if it is needed to complete a chain)
    destinationCACertificate: |-
      < Place the public CA certificate for the event processing cluster here.
      If no custom Issuer was supplied, this will be the ca value in the
      <your_ep_instance_name>-ss-cacert-kp secret. If a custom Issuer has been supplied,
      the signing CA for the supplied certificate chain will be needed. >
    insecureEdgeTerminationPolicy: None
  wildcardPolicy: None

Renewing certificates

Note the following points before you renew the TLS certificates for your Flink cluster: