IBM Support

Cloud Pak for Security: Unable to deploy Cases Elastic Search pod when Approval Strategy is set to Manual

Troubleshooting


Problem

Problem fully deploying the Cases Elastic Search pods in Cloud Pak for Security (CP4S) while OpenShift Lifecycle Manager (OLM) reports UpgradePending errors.

Symptom

 Cloud Pak for Security cluster status errors:
ERROR statefulsets isc-cases--a11a-bb-1aa1-es-server-data: replicas 1 out of 3 
ERROR: CR CP4SSoar soar has status Progressing: validation failed for Cases/cases value [status]False is not expected one for type=Successful
OpenShift Lifecycle Manager (OLM) errors:
export CP4S_NAMESPACE=<your-CP4S-namespace-here>
oc exec -ti deploy/cp-serviceability -n $CP4S_NAMESPACE -- /opt/bin/olm_check -- token "$(oc whoami -t)"

ERROR: Subscription cp-serviceability-operator: state: UpgradePending and has installed CSV cp-serviceability-operator.v1.10.28 while latest is cp-serviceability-operator.v1.10.29
(..)
ERROR: olm-operator: time="(..)" level=warning msg="needs reinstall: waiting for deployment ibm-mongodb-operator to become ready: deployment \"ibm-mongodb-operator\"
not available: Deployment does not have minimum availability." csv=ibm-mongodb-operator.v1.18.13 id=1aaaa namespace=ibm-common-services phase=Failed strategy=deployment
ERROR: olm-operator: (..) http: TLS handshake error from X.X.X.X:52694: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of "x509: invalid signature: parent certificate cannot sign this kind of certificate" while trying to verify candidate authority certificate "Red Hat, Inc.")

Cause

Cause 1:
There's a CP4S Operator that needs to be updated and this is impacting CP4S Elastic Search pods. In this case, we have the following:
  • cp-serviceability-operator with an UpgradePending status
  • OLM reports a Warning related to an IBM Cloud Pak Foundational Services ibm-mongodb-operator. This indicates the ClusterServiceVersion for this operator failed. 
  • OLM reports a "TLS handshake error" from an OPenShift resource. Notice, this is not related to CP4S.
Cause 2:
One of the Elastic Search pods didn't update the certificate after it expired and it was already renewed. 
These issues might not be exhibited at the same time.

Environment

  • Cloud Pak for Security 1.10.28 or newer
    • Approval Strategy has been set to Manual.
  • OpenShift Container Platform 4.16.38 or newer

Diagnosing The Problem

 Review the CP4S Cluster status:
export CP4S_NAMESPACE=<cp4s_namespace>
oc exec -ti deploy/cp-serviceability -n $CP4S_NAMESPACE -- /opt/bin/status --token "$(oc whoami -t)"
Check the OpenShift Lifecycle Manager (OLM) status:
oc exec -ti deploy/cp-serviceability -n $CP4S_NAMESPACE -- /opt/bin/olm_check -- token "$(oc whoami -t)"
 Review the Validity of the Elastic Search Certificate:
oc get secrets isc-cases-opensearch-ibm-elasticsearch-tls-secret -o jsonpath="{.data['tls\.crt']}" -n $CP4S_NAMESPACE| base64 -d | openssl x509 -noout -text

Resolving The Problem

Warning: The following procedure might update all of the CP4S components. Please, run these commands under an authorized maintenance window. 
  1. Backup environment.
  2. If you are seeing the UpgradePending errors reported by the OpenShift Lifecycle Manager, first, upgrade to a later CP4S release by following the appropriate documentation link page below:
    Note: If you are not seeing the OLM error skip steps 2 and 3 and jump to step 4.
  3. Wait for the upgrade process to finish. In case the upgrade fails, follow the steps to collect must-gather data and contact IBM Support for more assistance.
  4. Once there are no OpenShift Lifecycle Manager (OLM) errors. Verify ibm-elasticsearch certificate is not expired by reviewing the dates under Validity: 
    oc get secrets isc-cases-opensearch-ibm-elasticsearch-tls-secret -o jsonpath="{.data['tls\.crt']}" -n $CP4S_NAMESPACE| base64 -d | openssl x509 -noout -text
  5. Scale down the Elastic Search (ES) deployment: 
    oc scale sts isc-cases--e79b-ib-6fb9-es-server-data --replicas=0
  6. Wait for the ES pods to be terminated.
  7. Scale up the Elastic Search deployment: 
    oc scale sts isc-cases--e79b-ib-6fb9-es-server-data --replicas=3
  8. Validate the three replicas of the Elastic Search pods are Running and Ready: 
    oc get pods -l app.kubernetes.io/managed-by=ibm-elasticsearch -n $CP4S_NAMESPACE
NAME                                       READY   STATUS    RESTARTS   AGE
isc-cases--e79b-ib-6fb9-es-server-data-0   2/2     Running   0          20d
isc-cases--e79b-ib-6fb9-es-server-data-1   2/2     Running   0          14d
isc-cases--e79b-ib-6fb9-es-server-data-2   2/2     Running   0          20d

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB77","label":"Automation Platform"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSTDPP","label":"IBM Cloud Pak for Security"},"ARM Category":[{"code":"a8m3p000000F8yxAAC","label":"Cloud Pak for Security (CP4S)-\u003EPerformance"},{"code":"a8m0z0000001h8pAAA","label":"Support-\u003ECases"}],"ARM Case Number":"TS019619189","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"and future releases;1.10.0;1.11.0"}]

Document Information

Modified date:
26 August 2025

UID

ibm17238374

Page Feedback