Troubleshooting
Problem
The upgrade failed or is pending when upgrading the Cloud Pak operator or service. If you check the install plan, we can see some "install plan" are in failed status, and if you check the reason, it reports, "Job was active longer than specified deadline Reason: DeadlineExceeded."
Symptom
One or more "install plans" are in failed status. When describing the failed install plan, it reports similar information:
Reason: JobIncomplete
Status: True
Type: BundleLookupPending
Last Transition Time: 2022-03-16T09:15:37Z
Message: Job was active longer than specified deadline
Reason: DeadlineExceeded
Environment
IBM Cloud Pak for Data v4.0.x
Diagnosing The Problem
Run the command to get the install plans:
oc get installplan -A
Check the related install plan by
oc describe installplan install-6h4zt -n ibm-common-services
Check the Status
Reason: JobIncomplete
Status: True
Type: BundleLookupPending
Last Transition Time: 2022-03-16T09:15:37Z
Message: Job was active longer than specified deadline
Reason: DeadlineExceeded
Resolving The Problem
1. Get the names of any failing jobs and related config maps in the openshift-marketplace
oc get jobs,cm -n openshift-marketplace
Example Output:
NAME COMPLETIONS DURATION AGE
job.batch/00701ef4b3edf6a31523be94fd0c9b10ef9d7ab8d7d4657b738fbb2f1cd9c9c 1/1 11s 321d
job.batch/010ce5076702bc23682959e87165039edbfaafeee64a1aeb8aa3d31ba59ea18 1/1 10s 417d
job.batch/038fb916ef8c0cb77e9cdd489a691800f2398670191009838e28016be45cb10 1/1 16s 300d
job.batch/09d55c1d83be59138b65b5d1e1beea0926ed7d2b89ec7388dd93e72b6d89d15 0/1 6m8s 1d
...
NAME DATA AGE
configmap/00701ef4b3edf6a31523be94fd0c9b10ef9d7ab8d7d4657b738fbb2f1cd9c9c 2 321d
configmap/010ce5076702bc23682959e87165039edbfaafeee64a1aeb8aa3d31ba59ea18 2 417d
configmap/038fb916ef8c0cb77e9cdd489a691800f2398670191009838e28016be45cb10 8 300d
configmap/09d55c1d83be59138b65b5d1e1beea0926ed7d2b89ec7388dd93e72b6d89d15 6 30d
...
2. Delete all the jobs not completed 0/1
Example:
Example:
oc delete job.batch/09d55c1d83be59138b65b5d1e1beea0926ed7d2b89ec7388dd93e72b6d89d15 -n openshift-marketplace
job.batch "09d55c1d83be59138b65b5d1e1beea0926ed7d2b89ec7388dd93e72b6d89d15" deleted
3. Delete the corresponding config maps of the jobs not completed in openshift-marketplace
Example:
oc delete configmap/09d55c1d83be59138b65b5d1e1beea0926ed7d2b89ec7388dd93e72b6d89d15 -n openshift-marketplace
configmap "09d55c1d83be59138b65b5d1e1beea0926ed7d2b89ec7388dd93e72b6d89d15" deleted
3. Delete the failed install plan in ibm-common-services found using the steps in the Diagnostic section
oc delete ip faildedip -n ibm-common-services
4. Restart the OLM pod in openshift-operator-lifecycle-manager namespace by deleting the pod.
oc get pod -n openshift-operator-lifecycle-manager |grep olm
Example Output:
olm-operator-7bb65b5d67-pnsjd 1/1 Running 1 19h
oc delete pod olm-*** -n openshift-operator-lifecycle-manager
5. Restart the operand-deployment-lifecycle-manager(ODLM) in the ibm-common-services namespace
oc get pods -A |grep operand-deployment-lifecycle-manager
Example Output:
olm-operator-7bb65b5d67-pnsjd 1/1 Running 1 19h
oc delete pod olm-*** -n openshift-operator-lifecycle-manager
Verification:
After completing all the steps, check the new install plan status to see if it can start successfully and the operator is upgraded
oc get sub,csv,installplan -n *namespace*
Related Information
Document Location
Worldwide
[{"Type":"MASTER","Line of Business":{"code":"LOB10","label":"Data and AI"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSHGYS","label":"IBM Cloud Pak for Data"},"ARM Category":[{"code":"a8m3p000000UoRRAA0","label":"Administration-\u003EUpgrade"}],"ARM Case Number":"","Platform":[{"code":"PF040","label":"Red Hat OpenShift"}],"Version":"All Versions"},{"Type":"MASTER","Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SS8QTD","label":"IBM Cloud Pak for Integration"},"ARM Category":[{"code":"a8m0z0000001hogAAA","label":"Common Services"}],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"},{"Type":"MASTER","Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SS2JQC","label":"IBM Cloud Pak for Automation"},"ARM Category":[{"code":"a8m0z0000001iU9AAI","label":"Operate-\u003EBAI Install\\Upgrade\\Setup"}],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"},{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSTDPP","label":"IBM Cloud Pak for Security"},"ARM Category":[{"code":"a8m0z0000001h8uAAA","label":"Install or Upgrade"}],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]
Was this topic helpful?
Document Information
Modified date:
25 July 2023
UID
ibm16586054