Troubleshooting
Problem
WDP-DB2-0 pod is inconsistent state and it is unable to recover due to database got corrupted.
Symptom
Cause
Environment
Diagnosing The Problem
oc get pods -o wide | egrep -iv '1/1|2/2|3/3|4/4|completed'
oc logs wdp-db2-0
oc get nodes
oc get pv
oc get pvc
oc describe pods wdp-db2-0
oc get pvc $(oc describe pods wdp-db2-0 | grep ClaimName | awk '{ print $2 }' )
oc get pv $(oc get pvc --no-headers $(oc describe pods wdp-db2-0 | grep ClaimName | awk '{ print $2 }' ) | awk '{ print $3 }')
Resolving The Problem
Option 1:
Step 1: Stop and restart the db2 instance
Run the following command
oc rsh wdb-db2-0
su - db2inst1
db2 force application all
db2stop
db2start
db2 activate db ilgdb
db2 activate db bgdb
db2 activate db lineage
db2 activate db wfdb
exit
exit
Step 2.Delete the pods to restart
https://www.ibm.com/support/pages/how-resolve-sql1042c%C2%A0%C2%A0unexpected%C2%A0system%C2%A0error%C2%A0occurred%C2%A0upon-database-connection-after-unexpected-outage
Option 3: Rebuilding the wdp-db2-0 and database gets initialized, manual re-sync is required.
In this option, existing persistent storage will be deleted and recreated to fix the corrupted storage volume.
Scale down the wdp-db2-0 pod:
oc scale sts wdp-db2 --replicas=0
Delete the persistent storage volume
# delete pv
export PV=$(oc get pvc --no-headers $(oc describe pods wdp-db2-0 | grep ClaimName | awk '{ print $2 }' ) | awk '{ print $3 }')
oc delete pv $PV
# delete pvc
export PVC=$(oc describe pods wdp-db2-0 | grep ClaimName | awk '{ print $2 }' )
oc delete pvc $PVC
Scale back the wdp-db2-0 to 1
oc scale sts wdp-db2 --replicas=1
This will rebuild the wdp-db2-0 datatabase instance. You can check the status in pods log
oc logs wdp-db2-0 -f
Wait for the pods to come up. After that you need to resync the glossary terms, workflow
delete the following service pods:
wkc-glossary-service, wkc-workflow-service, wdp-policy-service, wdp-lineage
oc delete pods $(oc get pods --no-headers | egrep 'wkc-glossary-service|wdp-policy-service|wdp-lineage|wkc-workflow-service'|grep -iv "Completed")
Check the status of the rebuilt database and tables
oc rsh wdp-db2-0
su - db2inst1
db2 connect to ilgdb
db2 "select tabname from syscat.tables where tabschema = 'DB2INST1'"
Now you can notice the database is rebuilt.
Document Location
Worldwide
Document Information
Modified date:
25 January 2021
UID
ibm16407820