IT33179: KUBERNETES BACKUP SUPPORT BAAS-TRANSACTION-MANAGER POD FREQUENTLY RESTARTS CAUSING BACKUP AND RESTORE ERRORS

APAR status

• Closed as program error.

Error description

• The 10.1.6 version of Spectrum Protect Plus Kubernetes Backup
  Support has faulty liveness and readiness probes for the
  baas-transaction-manager pod that causes this pod to restart
  frequently.
  
  If you think you are experiencing this problem issue the
  following command, if the output is similar, you are
  experiencing this APAR.
  
  kubectl get events -n baas
  --sort-by=.metadata.creationTimestamp
  
  show these errors:
  
  55m Warning Unhealthy
  pod/baas-transaction-manager-bb6ff88bb-759lx Liveness probe
  failed: Get https://192.168.223.204:5000/: net/http: request
  canceled (Client.Timeout exceeded while awaiting headers)
  7m51s Warning Unhealthy
  pod/baas-transaction-manager-bb6ff88bb-759lx Readiness probe
  failed: Get https://192.168.223.204:5000/: net/http: request
  canceled (Client.Timeout exceeded while awaiting headers)
  54m Normal Killing pod/baas-transaction-manager-bb6ff88bb-759lx
  Container baas-transaction-manager-worker failed liveness probe,
  will be restarted
  54m Normal Pulling pod/baas-transaction-manager-bb6ff88bb-759lx
  Pulling image
  "9.11.70.74:5000/baas/baas-transaction-manager-worker:10.1.6"
  

Local fix

• baas-transaction-manager-worker has faulty liveness and rediness
  probes that can cause both baas-transaction-manager-worker and
  baas-transaction-manager to restarted repeatedly.
  
  To work around the this error, edit baas-k8s/baas/values.yaml
  file by updating readinessProbe and livenessProbe - enabled:
  false
  
  as shown below
  
  tmw:
  image:
  name: baas-transaction-manager-worker
  tag: 10.1.6
  pullPolicy: Always
  port: 5000
  portName: "tm-worker"
  readinessProbe:
  enabled: false
  initialDelaySeconds: 5
  periodSeconds: 10
  failureThreshold: 5
  livenessProbe:
  enabled: false
  initialDelaySeconds: 10
  periodSeconds: 20
  failureThreshold: 5
  
  and run ./baas-install.sh -u to update the configuration
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * IBM Spectrum Protect Plus level 10.1.6 Kubernetes Backup     *
  * Support                                                      *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * IBM Spectrum Protect Plus Kubernetes Backup Support v10.1.6  *
  * has abnormal restarts with the transaction-manager pod       *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fixing level when available. This problem is currently *
  * projected to be fixed in IBM Spectrum Protect Plus level     *
  * 10.1.7. Note that this is subject to change at the           *
  * discretion of IBM.                                           *
  ****************************************************************
  

Problem conclusion

• The code has been corrected to avoid causing the condition
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  SP PLUS

• Fixed component ID

  5737SPLUS

Applicable component levels