Overview

IBM® Automation OpenSearch is an open source search and analytics suite that can be used to ingest, search, visualize, and analyze data.

The ibm-elasticsearch-operator-ibm-es-controller-manager deployment provides the Opensearch APIs.

Prerequisites

Complete the following prerequisites to deploy the Opensearch:

Connection details for Opensearch

The Opensearch.status.conditions section returns the condition status, such as the Prerequisites indicates that the prerequisite Kubernetes resources are created, and the ElasticsearchSnapshotRepo indicates that an Elasticsearch Snapshot repository is successfully created. The Opensearch.status.url section returns the internal endpoint. The following is an example of the Opensearch custom resource:

apiVersion: elastic.automation.ibm.com/v1beta1
kind: Opensearch
...
status:
  conditions:
    - lastTransitionTime: '2024-06-13T07:07:46.822494Z'
      message: Successfully created Service Account
      reason: Success
      status: 'True'
      type: ServiceAccount
    - lastTransitionTime: '2024-06-13T07:07:54.524834Z'
      message: Credentials successfully created
      reason: Success
      status: 'True'
      type: Credentials
    - lastTransitionTime: '2024-06-13T07:08:01.613361Z'
      message: Certificate created successfully
      reason: Successful
      status: 'True'
      type: CertificateManager
    - lastTransitionTime: '2024-06-13T07:08:02.308896Z'
      message: Prerequisites have been deployed
      reason: Successful
      status: 'True'
      type: Prerequisites
    - lastTransitionTime: '2024-06-13T07:08:19.737445Z'
      message: PVC fields or labels are up to date
      reason: Updated
      status: 'True'
      type: UpdatePVC
    - lastTransitionTime: '2024-06-13T07:08:12.138102Z'
      message: ElasticsearchServer has successfully been deployed
      reason: Success
      status: 'True'
      type: ElasticsearchServer
    - lastTransitionTime: '2024-06-13T07:08:21.459349Z'
      message: 3  Elasticsearch pods are now available
      reason: Success
      status: 'True'
      type: ElasticsearchServerPods
    - lastTransitionTime: '2024-06-13T07:11:40Z'
      message: cloudpak snapshot repo is now available
      reason: Success
      status: 'True'
      type: ElasticsearchSnapshotRepo
    - lastTransitionTime: '2024-06-13T07:11:48.551109Z'
      message: Ready for searching...
      reason: Successful
      status: 'True'
      type: Started
  phase: Ready
  quiesce: false
  requiresRebuild: false
  url: 'https://opensearch-ibm-elasticsearch-srv'
  version: 1.1.2153
  versions:
    available:
      versions:
        - '1.3'
        - '2'
        - 6.8.22
        - 7.8.0
        - 7.8.1
        - 7.9.3
        - 7.10.2
        - haproxy
    reconciled: '2'

StorageClass

A ReadWriteOnce (RWO) PersistentVolume (PV) is needed for Opensearch. If you do not specify a StorageClass in the spec.nodes.storageClass section, the default StorageClass is used that you set for your cluster.

Storage

An RWX, shared PersistentVolumeClaim (PVC) for the Flink JobManagers and TaskManagers provides stateful checkpoint and savepoint for Flink jobs. When you have more than 1 Flink replica, the RWX PVC is mandatory.

For a single replica, you can mount the volume as read/write on a single node.

Make sure that you provision an RWX storage class in your cluster.

In the Opensearch custom resource, provide the following PVC as the volume jobs-storage:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: icp4adeploy-bai-pvc
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 20Gi
  storageClassName: ocs-storagecluster-cephfs

Example of Opensearch custom resource

An Opensearch custom resource with status is shown in the following example:

apiVersion: elasticsearch.opencontent.ibm.com/v1
kind: ElasticsearchCluster
metadata:
  annotations:
    cloudpakId: '12345'
    cloudpakName: CloudpakOpen
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"elasticsearch.opencontent.ibm.com/v1","kind":"ElasticsearchCluster","metadata":{"annotations":{"cloudpakId":"12345","cloudpakName":"CloudpakOpen","productChargedContainers":"All","productID":"8675309","productMetric":"VIRTUAL_PROCESSOR_CORE","productName":"CloudpakOpen
      Elasticsearch"},"name":"elasticsearchcluster-sample-tos","namespace":"es-os-ns"},"spec":{"addDefaultPlugins":true,"credentialSecret":"","image":"","imagePullSecret":"","maxUnavailable":0,"nodes":[{"addKeys":[],"additionalConfiguration":[{"name":"gateway.expected_data_nodes","value":"0"},{"name":"reindex.remote.allowlist","value":"[\"elasticsearch-cr-es-es-os-ns.apps.iaf-migration-414x.cp.fyre.ibm.com:443\"]"}],"data":true,"description":"All
      in one
      nodes","environmentVariables":{},"esResources":{"limits":{"cpu":"1000m","memory":"8Gi"},"requests":{"cpu":"100m","memory":"2Gi"}},"haResources":{"limits":{"cpu":"2000m","memory":"2Gi"},"requests":{"cpu":"100m","memory":"512Mi"}},"ingest":true,"master":true,"mountSecrets":[],"name":"all","nodeAffinity":{},"nodeGroupLabels":{},"nodeSelector":{},"podAffinity":{},"podAntiAffinity":{},"replicas":3,"secureEnvironmentVariables":[],"storageClass":"rook-cephfs","storageSize":"1"}],"odlmRegistry":"common-service","odlmRegistryNamespace":"ibm-common-services","proxyHttpPort":9200,"proxyTransportPort":9300,"serviceAccount":"","snapshotRepo":{"enabled":true,"size":"1Gi","snapshotActivity":"backup","storageClass":"rook-cephfs"},"tlsIssuer":"","tlsSecret":"","useCertificateManager":true,"useODLM":false,"version":"2"}}
    productChargedContainers: All
    productID: '8675309'
    productMetric: VIRTUAL_PROCESSOR_CORE
    productName: CloudpakOpen Elasticsearch
  name: elasticsearchcluster-sample
  namespace: es-os-ns
spec:
  addDefaultPlugins: true
  minimumMasterNodes: 2
  allowRebuildOnChange: false
  enableDataNodeService: false
  sharedStoragePVC: ''
  credentialSecret: ''
  odlmRegistryNamespace: ibm-common-services
  tlsSecret: ''
  maxUnavailable: 0
  imagePullSecret: ''
  nodes:
    - nodeSelector: {}
      jvmOpts: ''
      master: true
      environmentVariables: {}
      ingest: true
      schedulerName: ''
      nodeAffinity: {}
      mountSecrets:
        - name: elasticsearch-cr-elasticsearch-es-client-cert-kp
          path: /workdir/apps/elasticsearch/config/esos
      data: true
      name: all
      haResources:
        limits:
          cpu: 2000m
          ephemeral-storage: 6Gi
          memory: 2Gi
        requests:
          cpu: 100m
          ephemeral-storage: 10Mi
          memory: 512Mi
      podAffinity: {}
      additionalConfiguration:
        - name: gateway.expected_data_nodes
          value: '0'
        - name: reindex.remote.allowlist
          value: '["elasticsearch-cr-es-es-os-ns.apps.iaf-migration-414x.cp.fyre.ibm.com:443"]'
        - name: reindex.ssl.certificate_authorities
          value: '/workdir/apps/elasticsearch/config/esos/ca.crt'
        - name: reindex.ssl.certificate
          value: '/workdir/apps/elasticsearch/config/esos/tls.crt'
        - name: reindex.ssl.key
          value: '/workdir/apps/elasticsearch/config/esos/tls.key'
      podAntiAffinity: {}
      topologySpreadConstraints: []
      addKeys: []
      esResources:
        limits:
          cpu: 1000m
          ephemeral-storage: 5Gi
          memory: 8Gi
        requests:
          cpu: 100m
          ephemeral-storage: 10Mi
          memory: 2Gi
      secureEnvironmentVariables: []
      replicas: 3
      storageSize: '1'
      description: All in one nodes
      nodeGroupLabels: {}
      roles:
        - data
        - ingest
        - cluster_manager
      storageClass: rook-cephfs
  imagePullPolicy: IfNotPresent
  odlmRegistry: common-service
  tlsIssuer: ''
  useResourceRequestLimitsForJVMHeapRatio: 0.5
  quiesce: false
  useResourceRequestLimitsForJVMHeap: false
  snapshotRepo:
    enabled: true
    seLinuxOptionsLevels: 's0:c26,c20'
    size: 1Gi
    snapshotActivity: backup
    storageClass: rook-cephfs
  proxyHttpPort: 9200
  version: '2'
  serviceAccount: ''
  image: ''
  ignoreForMaintenance: false
  updateStrategy: RollingUpdate
  proxyTransportPort: 9300
  useODLM: false
  enableNetworkPolicy: true
  useCertificateManager: true
  sessionAffinity: ClientIP

{: codeblock}