Configuring OpenSearch custom resources

You can edit the OpenSearch custom resource (CR) configuration to reduce the OpenSearch PVC size from the default.

To reduce the PVC size after deployment, back up your data, delete existing PVCs, and restore and re-create the PVCs with updated sizeLimit values.

Reduced PVC sizing is generally acceptable for development and test environments with light workloads. However, before you reduce storage allocation, consider the following potential conditions if the disk becomes full:

• Index growth exceeds the allocated capacity.
• Clusters might become unstable under heavy write workloads.
• Proactive storage monitoring and capacity planning might be needed with reduced storage.

The ibm-openSearch-operator operator generates a default password for the OpenSearch admin user. However, you can specify your own custom Kubernetes Secret that contains the admin credentials. Customize admin credentials for enhanced security, compliance requirements, or to avoid including default credentials in production environments.

1. Create a Base64-encoded credential with the following command. In this example, opensearch@123 is the custom password for your admin user.

   echo -n 'opensearch@123' | base64

   The command results in output similar to the following example:

   b3BlbnNlYXJjaEAxMjM=

2. In the same namespace where the OpenSearch cluster is deployed, run the oc apply -f or kubectl apply -f command to create a custom Secret with an admin name and Base64-encoded password.

   kind: Secret
   apiVersion: v1
   metadata:
     name: custom-admin-user
     namespace: <namespace>
   data:
     opensearch-admin: b3BlbnNlYXJjaEAxMjM=
   type: Opaque

3. Reference the Secret in your OpenSearch CR.

   apiVersion: opensearch.cloudpackopen.ibm.com/v1
   kind: Cluster
   ...
   spec:
     ...
     plugins:
       ...
       security:
         enabled: true
         version: 3.2.0.0 # Replace this value with your latest OpenSearch version.
         internalUserSecret: custom-admin-user

   After the custom-admin-user Secret is configured, the OpenSearch route and indexes become accessible with the custom-admin-user credentials. The custom Secret is used to populate the internal_users file inside of the operand pods, and the operator doesn't generate the default admin:password admin credentials.

   If the internalUserSecret Secret is used to populate the internal_users.yml file, the credentials are available to all components that use internal user authentication. This authentication includes both route-based access and intra-cluster and internal client communication unless LDAP or another authentication domain takes precedence for internal access.

From IBM Cloud Pak® foundational services version 4.6.20 and onward, you can fetch custom admin credentials directly from HashiCorp Vault by using the Secrets Store CSI Driver. The ibm-opensearch-operator operator generates a default password for the admin user, admin:password. However, you can override this behavior by configuring Vault integration and specifying your own custom credentials that are stored in Vault.

From foundational services version 4.6.19 and onward, you can configure and enable audit logging in OpenSearch with a sidecar container. The sidecar converts audit logs into the Cloud Auditing Data Federation (CADF) format and optionally forwards them to the Zen audit service.

Enable forwarding to Zen according to your audit logging requirements. If your requirement is for OpenSearch audit logs to be generated in the CADF format and forwarded to Zen, similar to other platform components, enable this feature. This feature is for environments where centralized audit collection through Zen is required, and OpenSearch audit events must conform to the CADF standard.

1. Enable the webhook audit configuration in the opensearchyml section.
2. Enable the audit sidecar in the audit section. In your OpenSearch CR YAML file, add the following parameter details under the spec.nodePools section.
   • Use the opensearchyml section to configure OpenSearch to send audit logs with the webhook. Replace the <CR_NAME> value with your cluster and the <NAMESPACE_NAME> value with your namespace.
   • Set the audit.enabled value to true to create and enable the audit sidecar container.
   • Set the audit.forwardToZen value to true if you want the sidecar to forward CADF audit logs to the Zen service.
   • The audit.resources value defines CPU and memory requests and limits for the audit sidecar container.
   • The audit.zenAuditURL Zen endpoint forwards the logs when the forwardToZen value is enabled.
   • The audit.zenSecretName value is the name of the Kubernetes Secret that contains the Zen token.

   apiVersion: opensearch.cloudpackopen.ibm.com/v1
   kind: Cluster
   metadata:
     name: opensearch-cr
     namespace: <namespace>
     ...
   spec:
     ...
     nodePools:
       ...
       opensearchyml: |
         plugins.security.audit.type: webhook
         plugins.security.audit.config.webhook.url: "http://<CR_NAME>.<NAMESPACE_NAME>.svc.cluster.local:8085/audit"
         plugins.security.audit.config.webhook.format: JSON
         plugins.security.audit.config.webhook.ssl.verify: false
       ...
       audit:
         enabled: true
         forwardToZen: <true/false>
         resources:
           limits:
             cpu: 200m
             memory: 256Mi
           requests:
             cpu: 50m
             memory: 64Mi
         zenAuditURL: '<ZEN_AUDIT_URL>'
         zenSecretName: <ZEN_TOKEN_SECRET_NAME>

3. If forwarding to Zen is enabled, create a Kubernetes Secret with the Zen token.

   kind: Secret
   apiVersion: v1
   metadata:
     name: <zen-audit-token-secret_name>
     namespace: <NAMESPACE>
   data:
     token: <ZEN_TOKEN in Base64 format>
   type: Opaque

For custom logging and log file management, configure a custom log4j2.properties file to enable OpenSearch log rotation.

1. Create a ConfigMap with your custom log4j2.properties file.

   The following configuration enables daily log rotation. If a file exceeds 1 MB, a maximum of two files are retained. Excess logs are automatically deleted if the total size exceeds 2 GB.

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: log4j-rotation-config
     namespace: <namespace>
   data:
     log4j2.properties: |-
       status = error
   
       # Console appender
       appender.console.type = Console
       appender.console.name = console
       appender.console.layout.type = PatternLayout
       appender.console.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] %m%n
   
       # Rolling file appender
       appender.rolling.type = RollingFile
       appender.rolling.name = rolling
       appender.rolling.fileName = logs/opensearch.log
       appender.rolling.filePattern = logs/opensearch-%d{yyyy-MM-dd}-%i.log.gz
       appender.rolling.layout.type = PatternLayout
       appender.rolling.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] %m%n
   
       appender.rolling.policies.type = Policies
       appender.rolling.policies.time.type = TimeBasedTriggeringPolicy
       appender.rolling.policies.time.interval = 1
       appender.rolling.policies.time.modulate = true
       appender.rolling.policies.size.type = SizeBasedTriggeringPolicy
       appender.rolling.policies.size.size = 1MB
   
       appender.rolling.strategy.type = DefaultRolloverStrategy
       appender.rolling.strategy.max = 2
       appender.rolling.strategy.fileIndex = nomax
       appender.rolling.strategy.action.type = Delete
       appender.rolling.strategy.action.basepath = logs
       appender.rolling.strategy.action.condition.type = IfAccumulatedFileSize
       appender.rolling.strategy.action.condition.exceeds = 2GB
   
       # Root logger
       rootLogger.level = info
       rootLogger.appenderRef.console.ref = console
       rootLogger.appenderRef.rolling.ref = rolling

2. Reference the ConfigMap in the OpenSearch CR. Disable the default Log4j configuration mount and mount your custom ConfigMap.

   apiVersion: opensearch.cloudpackopen.ibm.com/v1
   kind: Cluster
   metadata:
     name: opensearch-cr
     namespace: <namespace>
     ...
   spec:
     ...
     nodePools:
       - disableConfigMounts:
           log4j: true  # Disable the default Log4j ConfigMap mounting with the operator.
         ...
         addConfigmaps:
           - location: /workdir/opensearch/config/log4j2.properties
             name: log4j-rotation-config    # Add your custom Log4j ConfigMap.
             subPath: log4j2.properties
   ...

For workload-specific JVM tuning or for troubleshooting, override default JVM options with your own jvm.options file in the OpenSearch cluster that is deployed with the IBM OpenSearch operator.

1. Create a ConfigMap with your custom JVM options.

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: jvm-custom-config
     namespace: <namespace>
   data:
     jvm.options: |-
       -Xms4g
       -Xmx4g
       -XX:+UseG1GC
       -XX:InitiatingHeapOccupancyPercent=75
       ...

2. Reference the custom JVM options in the CR. Like with the log4j2.properties file, disable the default jvm.options file and mount your custom file.

   apiVersion: opensearch.cloudpackopen.ibm.com/v1
   kind: Cluster
   metadata:
     name: opensearch-cr
     namespace: <namespace>
     ...
   spec:
     ...
     nodePools:
       - name: all
         disableConfigMounts:
           jvmOptions: true    # Disable the default jvmOptions ConfigMap mounting with the operator.
         addConfigmaps:
           - name: jvm-custom-config
             location: /workdir/opensearch/config/jvm.options    # Add your custom jvm.options ConfigMap.
             subPath: jvm.options
   ...

The plugins.security.ssl.http.clientauth_mode parameter controls whether OpenSearch requires client certificates during HTTP connections that are used by REST APIs. Use this setting to enable mutual Transport Layer Security (mTLS) authentication for REST.

For more information about TLS client authentication modes, see Client authentication .