IT34128: /VAR/LOG/MESSAGES FILLED WITH ERRORS "PULL ACCESS DENIED" ABOUT DOCKER IMAGES

APAR status

• Closed as program error.

Error description

• Starting with version 10.1.6, the IBM Spectrum Protect Plus
  host has a Kubernetes (k8s) implementation to support backups
  for Amazon Elastic Compute Cloud (EC2) workloads.
  That
  implementation includes the deployment of several docker images
  that come with the installation package.
  If the file system
  hosting the directory /var/lib/docker is running out of space
  for any reason, the k8s Garbage Collector will start deleting
  some of these images.
  When that happens, the missing image(s)
  are detected which triggers the system to get these back and
  then fails throwing the following example message to
  /var/log/messages.
  
  <timestamp> <SPPHostName> kubelet[1317]:
  E0702 <timestamp> xxxx remote_image.go:113] PullImage
  "ibm/spp-mongo:4.2.0" from image service failed: rpc error:
  code = Unknown desc = Error response from daemon: pull access
  denied for <DockerImage>, repository does not exist or may
  require 'docker login': denied: requested access to the
  resource is denied
  
  Since this is retried indefinitely, it
  fills up the /var/log/messages file.
  
  IBM Spectrum Protect Plus
  Versions Affected:
  IBM Spectrum Protect Plus 10.1.6
  
  Initial
  Impact: Medium
  
  Additional Keywords: SPP, SPPLUS, TS003894577,
  docker, Kubernetes, spam, filling
  

Local fix

• Increase the available space for /var/lib/docker
  OR
  If no
  Amazon EC2 backups are needed or planned, k8s services can be
  disabled to avoid the /var/log/messages filling up.
  The backup
  of Kubernetes clients does not use the on-board k8s
  services.
  This is done with the following set of commands on
  the IBM Spectrum Protect host :
   sudo systemctl stop
  k8s-plugins
   sudo systemctl disable k8s-plugins
   sudo touch
  /opt/SPP/spp-k8s/reset_cluster
   sudo kubeadm reset -f
  
  If for
  any reason the k8s service needs to be enabled again run the
  following command :
   sudo rm /opt/SPP/spp-k8s/reset_cluster
  
  sudo systemctl enable k8s-plugins
   sudo systemctl start
  k8s-plugin
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * IBM Spectrum Protect Plus level 10.1.6.                      *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See Error Description.                                       *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply the fixing level when available. This problem was      *
  * fixed in IBM Spectrum Protect Plus level 10.1.7. Note that   *
  * this is subject to change at the discretion of IBM.          *
  ****************************************************************
  

Problem conclusion

• The memory requirement for plugins has been adjusted, so there
  is no need to disable the plugins.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  SP PLUS

• Fixed component ID

  5737SPLUS

Applicable component levels