ping command fails in Db2 cluster
Upgrade to IBM Software Hub Version 5.1 before IBM Cloud Pak for Data Version 4.5 reaches end of support. For more information, see Upgrading IBM Software Hub in the IBM Software Hub Version 5.1 documentation.
After deploying Db2 in a db2ucluster, the ping command fails. This applies to IBM Cloud Pak for Data 4.5.1 and 4.5.2 on Red Hat® OpenShift® Kubernetes Service (ROKS). Addition of a kernel parameter might be required.
Symptoms
- A Db2 pod remains in a
0/1Ready state. - After running the
oc logs -f <db2u_pod>command on a Db2 pod with a-db2u-0suffix, or-db2u-0,-db2u-1,-db2u-(n+1)suffixes in Db2® Warehouse MPP, the followingping: socket: Operation not permittedmessage is found in the pod log:+ echo 'Command failed. Attempt 120/120:' Command failed. Attempt 120/120: + sleep 5 + true + timeout 1 ping -c 1 c-db2oltp-db2u-0.c-db2oltp-db2u-internal ping: socket: Operation not permitted + [[ 120 -lt 120 ]] + fail 'The command has failed after 120 attempts.' + echo The command has failed after 120 attempts. The command has failed after 120 attempts.
Resolving the problem
- Find the respective db2ucluster resource name and create a
db2uclusterenvironment variable:oc get db2ucluster --all-namespacesdb2ucluster=db2ucluster_resource_name - Configure
namespaceandstatefulsetvariables:namespace=$(oc get db2ucluster --all-namespaces | grep ${db2ucluster} | awk {'print $1'})statefulset=$(oc get sts -n ${namespace} | grep ${db2ucluster}-db2u | awk {'print $1'}) - To determine if the db2ucluster deployment is
restricted: true, run the following command:oc get db2ucluster ${db2ucluster} -o yaml | grep -i restricted
The command returns restricted: true
Run the following patch command:oc exec -it ${statefulset}-0 -- /bin/bash -c "touch /db2u/tmp/.pause_probe"
The command returns restricted: false or nothing
Proceed to the next step.
- Wait for the formation status of the Db2 StatefulSet to become
OKbefore proceeding to the next step. To obtain the status, run the following command:
Look for the following result:oc get formations.db2u.databases.ibm.com ${db2ucluster} -n ${namespace} -o go-template='{{range .status.components}}{{printf "%s,%s,%s\n" .kind .name .status.state}}{{end}}' | column -s, -tStatefulSet ${db2ucluster}-db2u OK - There are two different commands to patch the StatefulSet. To determine which command to use,
run the following command:
oc get sts ${statefulset} -o yaml | grep sysctl
The command does not return an output
Run the following patch command:oc patch sts ${statefulset} -n ${namespace} -p '{"spec": {"template":{"spec":{"securityContext":{"sysctls":[{"name": "net.ipv4.ping_group_range","value": "0 2147483647"}]}}}}}'
The command returns an output such as sysctls: and nothing else
Run the following patch command:oc patch sts ${statefulset} -n ${namespace} --type json -p '[{"op":"add","path":"/spec/template/spec/securityContext/sysctls/-","value":{"name":"net.ipv4.ping_group_range","value":"0 2147483647"}}]'
The
db2u-0pod will enter a terminating state and restart. Wait for thedb2u-0pod to become1/1Running state before proceeding to the next step. - To ensure that the
db2u-0pod completed the entry point, the end of the pod log includes a timestamp similar to the following:========================================= ### STARTTIME=20220817154656 ###" ### ENDTIME=20220817155051 ###" ### TIMEDIFF=395 s ###" ========================================= + exit 0
1/1 Running state, examine the output of the respective
sts yaml file by running the following
command:oc get sts ${statefulset} -n ${namespace} -o yaml | grep "net.ipv4.ping_group_range"Look
for the following result:- name: net.ipv4.ping_group_range
sysctls:
- name: net.ipv4.ping_group_range
value: 0 2147483647
The db2u pods under the StatefulSet resource should proceed beyond the initial
ping command failure.Additional steps for InfoSphere Information Server (IIS) and Watson Knowledge Catalog (WKC) deployments
If your db2ucluster resource was created from an IIS or WKC custom resource, the extra databases
might be missing. WKC pod c-db2oltp-wkc-db2u-0 might contain only the
LINEAGE database and IIS pod c-db2oltp-iis-db2u-0 might contain
only the XMETA database.
- Symptoms
- Pod with prefix
wkc-db2u-initmight remain in theRunningorErrorstate and never reach theCompletedstate. Pod with prefixiis-db2u-backup-restore-jobmight remain in theErrorstate.
- Remove the
"addDatabaseTriggered":truetext by running the following command:oc edit cm db2aaservice-databases-watch-cmFor example,
becomes[{"addDatabase":true,"addDatabaseTriggered":true,"databaseConfigurationOverrides": {"db-cfg":{"BGDB":{"AUTHN_CACHE_USERS":"10","DFT_EXTENT_SZ":"8","LOGARCHMETH1":"OFF", "LOGFILSIZ":"10240","LOGSECOND":"128"},"ILGDB":{"DFT_EXTENT_SZ":"256","LOGARCHMETH1":"OFF", "LOGFILSIZ":"10240","LOGSECOND":"100"},"LINEAGE":{"DFT_EXTENT_SZ":"256","LOGARCHMETH1":"OFF"}, "WFDB":{"DFT_EXTENT_SZ":"8","LOGARCHMETH1":"OFF","LOGSECOND":"100"}}},"dbname": "LINEAGE,BGDB,ILGDB,WFDB","dbtype":"db2oltp","instanceId":"db2oltp-wkc","namespace":"wkc"}, {"addDatabase":true,"addDatabaseTriggered":true,"databaseConfigurationOverrides": {"db-cfg":{"DSODB":{"LOGARCHMETH1":"OFF","LOGFILSIZ":"1000","LOGPRIMARY":"50", "LOGSECOND":"200"},"IADB":{"LOGARCHMETH1":"OFF","LOGFILSIZ":"15000","LOGPRIMARY":"50", "LOGSECOND":"200"},"XMETA":{"LOGARCHMETH1":"OFF","LOGFILSIZ":"10000","LOGPRIMARY":"50", "LOGSECOND":"200"}}},"dbname":"XMETA,IADB,DSODB","dbtype":"db2oltp","instanceId": "db2oltp-iis","namespace":"wkc"}][{"addDatabase":true, "databaseConfigurationOverrides":{"db-cfg":{"BGDB": {"AUTHN_CACHE_USERS":"10","DFT_EXTENT_SZ":"8","LOGARCHMETH1":"OFF","LOGFILSIZ":"10240", "LOGSECOND":"128"},"ILGDB":{"DFT_EXTENT_SZ":"256","LOGARCHMETH1":"OFF","LOGFILSIZ":"10240", "LOGSECOND":"100"},"LINEAGE":{"DFT_EXTENT_SZ":"256","LOGARCHMETH1":"OFF"},"WFDB": {"DFT_EXTENT_SZ":"8","LOGARCHMETH1":"OFF","LOGSECOND":"100"}}},"dbname":"LINEAGE,BGDB, ILGDB,WFDB","dbtype":"db2oltp","instanceId":"db2oltp-wkc","namespace":"wkc"},{"addDatabase":true, "databaseConfigurationOverrides":{"db-cfg":{"DSODB":{"LOGARCHMETH1":"OFF","LOGFILSIZ":"1000", "LOGPRIMARY":"50","LOGSECOND":"200"},"IADB":{"LOGARCHMETH1":"OFF","LOGFILSIZ":"15000", "LOGPRIMARY":"50","LOGSECOND":"200"},"XMETA":{"LOGARCHMETH1":"OFF","LOGFILSIZ":"10000", "LOGPRIMARY":"50","LOGSECOND":"200"}}},"dbname":"XMETA,IADB,DSODB","dbtype":"db2oltp", "instanceId":"db2oltp-iis","namespace":"wkc"}]The missing databases will now provision and the
wkc-db2u-initpod will enter theCompletedstate after the databases complete provisioning. - If the
iis-db2u-backup-restore-jobpod persists in theErrorstate, restart the job by completing the following steps:- Delete the
job:
oc delete job -n ${namespace} iis-db2u-backup-restore-job - Restart the
ibm-cpd-iis-operatoroperator by running the following command:oc get po -A -l app.kubernetes.io/name=ibm-cpd-iis-operatorThe command returns the pod and namespace to delete that is similar to the following example result:NAMESPACE NAME READY STATUS RESTARTS AGE ibm-common-services ibm-cpd-iis-operator-5c788487cc-ncpz6 1/1 Running 0 119mTo delete the returned example pod and namespace, run the following example command:oc delete pod ibm-cpd-iis-operator-5c788487cc-ncpz6 -n ibm-common-services
- Delete the
job: