LI83089: POSTGRES PODS RESTARTS FREQUENTLY DUE TO LIVENESS TIMEOUT

APAR status

• Closed as program error.

Error description

• Postgres pods restart frequently. The describe of the pod
  complains of timeout error:
  
  Liveness probe failed: command
  "/opt/crunchy/bin/postgres-ha/health/pgha-liveness.sh" timed out
  
  The version of yq delivered with out database vendor - version
  3.3.0 has a known issue which is causing yq to take longer to
  run, and eventually times out.  Fix is to increase the liveness
  timeout of postgres pods from 10 seconds to 30 seconds.
  

Local fix

• 1) For all the postgres deployments, edit timeoutSeconds from 10
  to 30 for liveness probe
  
  Steps:
  -Find the name of the deployments using
  kubectl get deploy | grep postgres
  - Edit the deployment
  kubectl edit deploy
  -Search for liveness, should look similar to below:
  
  livenessProbe:          exec:
             command:
             -
  /opt/crunchy/bin/postgres-ha/health/pgha-liveness.sh
           failureThreshold: 3
           initialDelaySeconds: 30
           periodSeconds: 15
           successThreshold: 1
           timeoutSeconds: 10
  
  -Edit the timeoutSeconds: 10 to timeoutSeconds: 30
  

Problem summary

• Postgres pods restart frequently. The describe of the pod
  complains of timeout error:
  
  Liveness probe failed: command
  "/opt/crunchy/bin/postgres-ha/health/pgha-liveness.sh" timed out
  

Problem conclusion

• Fix is added in APIC release v10.0.5.6. Increased the liveness
  timeout of postgres pods from 10 seconds to 30 seconds.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  API CONNECT ENT

• Fixed component ID

  5725Z2201

Applicable component levels