Overview

IBM® Automation OpenSearch is an open source search and analytics suite that can be used to ingest, search, visualize, and analyze data.

The ibm-elasticsearch-operator-ibm-es-controller-manager deployment provides the Opensearch APIs.

• Prerequisites
• Connection details for Opensearch
• StorageClass
• Example Opensearch CR

Prerequisites

Complete the following prerequisites to deploy the Opensearch:

• Ensure that the ibm-cert-manager-operator or cert-manager is installed on the cluster if the useCertificateManager parameter is set to true in the OpensearchCluster custom resource.

• Ensure that you install the Opensearch in the namespace where you deployed the foundational services.

Connection details for Opensearch

The Opensearch.status.endpoints section returns connection details, such as the Secret with the administrator credentials, and the internal and external endpoints, when you request for them. The following is an example section from the Opensearch custom resource:

apiVersion: elastic.automation.ibm.com/v1beta1
kind: Opensearch
...
status:
  endpoints:
    - authentication:
        secret:
          secretName: iaf-system-opensearch-es-default-user
        type: BasicSecret
      caSecret:
        key: ca.crt
        secretName: [automationbase-name]-automationbase-ab-ca
      name: iaf-system-es
      scope: External
      type: API
      uri: 'https://iaf-system-es-iaf.apps.iaf-test2.cp.fyre.ibm.com'
    - authentication:
        secret:
          secretName: iaf-system-opensearch-es-default-user
        type: BasicSecret
      caSecret:
        key: ca.crt
        secretName: automationbase-sample-automationbase-ab-ca
      name: iaf-system-opensearch-es
      scope: Internal
      type: API
      uri: 'https://iaf-system-opensearch-es.iaf:9200'

StorageClass

A ReadWriteOnce (RWO) PersistentVolume (PV) is needed for Opensearch. If you do not specify a StorageClass in the spec.nodes.storageClass section, the default StorageClass is used that you set for your cluster.

Storage

An RWX, shared PersistentVolumeClaim (PVC) for the Flink JobManagers and TaskManagers provides stateful checkpoint and savepoint for Flink jobs. When you have more than 1 Flink replica, the RWX PVC is mandatory.

For a single replica, you can mount the volume as read/write on a single node.

Note: Make sure that you provision an RWX storage class in your cluster.

In the Opensearch custom resource, provide the following PVC as the volume jobs-storage:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: icp4adeploy-bai-pvc
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 20Gi
  storageClassName: ocs-storagecluster-cephfs

Example of Opensearch custom resource

An Opensearch custom resource with status is shown in the following example:

apiVersion: elasticsearch.opencontent.ibm.com/v1
kind: ElasticsearchCluster
metadata:
  annotations:
    cloudpakId: '12345'
    cloudpakName: CloudpakOpen
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"elasticsearch.opencontent.ibm.com/v1","kind":"ElasticsearchCluster","metadata":{"annotations":{"cloudpakId":"12345","cloudpakName":"CloudpakOpen","productChargedContainers":"All","productID":"8675309","productMetric":"VIRTUAL_PROCESSOR_CORE","productName":"CloudpakOpen
      Elasticsearch"},"name":"elasticsearchcluster-sample-tos","namespace":"es-os-ns"},"spec":{"addDefaultPlugins":true,"credentialSecret":"","image":"","imagePullSecret":"","maxUnavailable":0,"nodes":[{"addKeys":[],"additionalConfiguration":[{"name":"gateway.expected_data_nodes","value":"0"},{"name":"reindex.remote.allowlist","value":"[\"elasticsearch-cr-es-es-os-ns.apps.iaf-migration-414x.cp.fyre.ibm.com:443\"]"}],"data":true,"description":"All
      in one
      nodes","environmentVariables":{},"esResources":{"limits":{"cpu":"1000m","memory":"8Gi"},"requests":{"cpu":"100m","memory":"2Gi"}},"haResources":{"limits":{"cpu":"2000m","memory":"2Gi"},"requests":{"cpu":"100m","memory":"512Mi"}},"ingest":true,"master":true,"mountSecrets":[],"name":"all","nodeAffinity":{},"nodeGroupLabels":{},"nodeSelector":{},"podAffinity":{},"podAntiAffinity":{},"replicas":3,"secureEnvironmentVariables":[],"storageClass":"rook-cephfs","storageSize":"1"}],"odlmRegistry":"common-service","odlmRegistryNamespace":"ibm-common-services","proxyHttpPort":9200,"proxyTransportPort":9300,"serviceAccount":"","snapshotRepo":{"enabled":true,"size":"1Gi","snapshotActivity":"backup","storageClass":"rook-cephfs"},"tlsIssuer":"","tlsSecret":"","useCertificateManager":true,"useODLM":false,"version":"2"}}
    productChargedContainers: All
    productID: '8675309'
    productMetric: VIRTUAL_PROCESSOR_CORE
    productName: CloudpakOpen Elasticsearch
  name: elasticsearchcluster-sample
  namespace: es-os-ns
spec:
  addDefaultPlugins: true
  minimumMasterNodes: 2
  allowRebuildOnChange: false
  enableDataNodeService: false
  sharedStoragePVC: ''
  credentialSecret: ''
  odlmRegistryNamespace: ibm-common-services
  tlsSecret: ''
  maxUnavailable: 0
  imagePullSecret: ''
  nodes:
    - nodeSelector: {}
      jvmOpts: ''
      master: true
      environmentVariables: {}
      ingest: true
      schedulerName: ''
      nodeAffinity: {}
      mountSecrets:
        - name: elasticsearch-cr-elasticsearch-es-client-cert-kp
          path: /workdir/apps/elasticsearch/config/esos
      data: true
      name: all
      haResources:
        limits:
          cpu: 2000m
          ephemeral-storage: 6Gi
          memory: 2Gi
        requests:
          cpu: 100m
          ephemeral-storage: 10Mi
          memory: 512Mi
      podAffinity: {}
      additionalConfiguration:
        - name: gateway.expected_data_nodes
          value: '0'
        - name: reindex.remote.allowlist
          value: '["elasticsearch-cr-es-es-os-ns.apps.iaf-migration-414x.cp.fyre.ibm.com:443"]'
        - name: reindex.ssl.certificate_authorities
          value: '/workdir/apps/elasticsearch/config/esos/ca.crt'
        - name: reindex.ssl.certificate
          value: '/workdir/apps/elasticsearch/config/esos/tls.crt'
        - name: reindex.ssl.key
          value: '/workdir/apps/elasticsearch/config/esos/tls.key'
      podAntiAffinity: {}
      topologySpreadConstraints: []
      addKeys: []
      esResources:
        limits:
          cpu: 1000m
          ephemeral-storage: 5Gi
          memory: 8Gi
        requests:
          cpu: 100m
          ephemeral-storage: 10Mi
          memory: 2Gi
      secureEnvironmentVariables: []
      replicas: 3
      storageSize: '1'
      description: All in one nodes
      nodeGroupLabels: {}
      roles:
        - data
        - ingest
        - cluster_manager
      storageClass: rook-cephfs
  imagePullPolicy: IfNotPresent
  odlmRegistry: common-service
  tlsIssuer: ''
  useResourceRequestLimitsForJVMHeapRatio: 0.5
  quiesce: false
  useResourceRequestLimitsForJVMHeap: false
  snapshotRepo:
    enabled: true
    seLinuxOptionsLevels: 's0:c26,c20'
    size: 1Gi
    snapshotActivity: backup
    storageClass: rook-cephfs
  proxyHttpPort: 9200
  version: '2'
  serviceAccount: ''
  image: ''
  ignoreForMaintenance: false
  updateStrategy: RollingUpdate
  proxyTransportPort: 9300
  useODLM: false
  enableNetworkPolicy: true
  useCertificateManager: true
  sessionAffinity: ClientIP

{: codeblock}