isc-profile or isc-auth pods fail during upgrade

When you upgrade from version 1.10.19 or later to version 1.10.23 or later, the isc-profile or isc-auth pods can appear to be in a crashloop state.

Symptoms

The isc-profile or isc-auth pods are in a crashloop state when a race condition occurs due to the order in which the Qradar Suite operator allocates the microservice custom resources.

The following example outputs show that the isc-profile and isc-auth pods are in a crashloop state.
oc get pod -A | grep -i crash 
cp4s isc-profile-7754d7559d-7j6p8 0/1 CrashLoopBackOff 5 (104s ago) 15m 
cp4s isc-profile-7754d7559d-rmnvv 0/1 CrashLoopBackOff 5 (37s ago) 16m
oc get pod -A | grep -i crash 
cp4s isc-auth-7754d7559d-rmnvv 0/1 CrashLoopBackOff 2 (37s ago) 16m
cp4s isc-auth-7754d7559d-7j6p8 0/1 CrashLoopBackOff 3 (37s ago) 16m

Resolving the problem

  1. Delete the isc-profile and isc-auth pods by using the following command. 
    oc delete iscguard iscprofile-app iscauth-app
    The following example output shows that the isc-profile and isc-auth pods are deleted.
    iscguard.isc.ibm.com "iscprofile-app" deleted
iscguard.isc.ibm.com "iscauth-app" deleted
  2. Delete the platform operator pod by using the following command.
     oc delete pod -lname=cp4s-platform-operator
    The following example output shows that the platform operator pod is deleted.
    pod "cp4s-platform-operator-85d6556464-cv5rs" deleted
  3. Verify that the pods are in a running state by using the following commands.
    oc get pod -lname=isc-profile -n $CP4S_NAMESPACE
    oc get pod -lname=isc-auth -n $CP4S_NAMESPACE
    The following example outputs show that the isc-profile and isc-auth pods are in a running state.
    oc get pod -lname=isc-profile -n cp4s
NAME                          READY   STATUS    RESTARTS   AGE
isc-profile-8f76ccc56-6t476   1/1     Running   0          48m
isc-profile-8f76ccc56-hlzhx   1/1     Running   0          49m
    oc get pod -lname=isc-auth -n cp4s 
NAME                      READY   STATUS    RESTARTS   AGE
isc-auth-f5b7b995-dczzx   1/1     Running   0          54m
isc-auth-f5b7b995-gxtcs   1/1     Running   0          53m