IBM Support

Cloud Pak for Security: Redis and statefulsets not present in namespaces

Troubleshooting


Problem

After a successful software upgrade,  redis pods and statefulset might not be present in Cloud Pak for Security (CP4S).
Not allowing users to access the tool since redis is not communicating with isc-entitlement pods as result showing a message after authenticating:
upstream request time out

Diagnosing The Problem

  1. In order to diagnose this issue, run these steps in the CP4S namespace: 
    oc login
oc project CP4S
    NOTE: Replace the CP4S variable with the name of your project.
  2.  List redis pods ibm-cloud-databases-redis-operator is the only pod running:  
    oc get pods | grep redis
  3. List statefulsets and check  statefulsets for redis are not present:  
    oc get statefulsets | grep redis
  4. Check the status of redis iscinventory object: 
    oc get iscinventory redis-default -o yaml | grep -A 10 ^status:
    Status:  
    status:
  conditions:
  - lastTransitionTime: "2022-03-29T22:48:57Z"
    message: 'dial tcp: lookup c-default-redis-p.cp4s.svc.cluster.local on 172.1.2.3:53:
       no such host'
    reason: Unexpected error
    status: "True"
    type: Failed

Resolving The Problem

Refreshing the inventory is enough to bring back redis pods online:
  1. In order to diagnose this issue, run these steps in the CP4S namespace: 
    oc login
oc project CP4S
    NOTE: Replace the CP4S variable with the name of your project.
  2. Delete redis operator:  
    oc get pod -o name | grep redis | xargs oc delete
  3. Once the redis operator is running, check the redissetinel status: 
    oc get redissentinel -o yaml | tail -15
    Look at message for synchronization is in progress: 
        version: 5.0.9
  status:
    conditions:
    - lastHeartbeatTime: "2022-07-14T13:28:52Z"
      lastTransitionTime: "2022-02-22T04:02:34Z"
      message: Synced waiting for formation resources to sync
      reason: SyncInProgress
      status: "Flase"
      type: Available
    formationGeneration: 3
    observedGeneration: 0

kind: List
metadata:
  resourceVersion: ""
  selfLink: ""
  4. Proceed deleting redissentinel and cp4s-helper-operator: 
    oc delete redissentinel --all
oc delete pod -lname=cp4s-helper-operator
  5. Wait for three to five minutes and to check the redissentinel status: 
    oc get redissentinel -o yaml | tail -15
    version: 5.0.9
  status:
    conditions:
    - lastHeartbeatTime: "2022-07-14T13:28:52Z"
      lastTransitionTime: "2022-02-22T04:02:34Z"
      message: Synced successfully
      reason: Synced
      status: "True"
      type: Available
    formationGeneration: 3
    observedGeneration: 722
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""
  6. Check redis pods status: 
    oc get pods | grep redis
c-default-redis-m-0                                               4/4     Running             0                  99s
c-default-redis-m-1                                               4/4     Running             0                  99s
c-default-redis-m-2                                               4/4     Running             0                  99s
c-default-redis-s-0                                               4/4     Running             0                  99s
c-default-redis-s-1                                               4/4     Running             0                  98s
c-default-redis-s-2                                               4/4     Running             0                  99s

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSTDPP","label":"IBM Cloud Pak for Security"},"ARM Category":[{"code":"a8m0z0000001h8uAAA","label":"Install or Upgrade"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]

Document Information

Modified date:
04 August 2022

UID

ibm16603767

Page Feedback