Cannot push new images to IBM Cloud Private

You cannot push a new image from an existing namespace on your local drive to your IBM® Cloud Private image registry.

Scenario 1: Cannot push an image to a private registry with a https endpoint

Scenario 2: NFS storage is configured as a registry backend storage

Scenario 1: Cannot push an image to a private registry with a https endpoint

Symptoms

As you log on to your IBM Cloud Private cluster, you cannot push new images from your local drive with an https endpoint. Your terminal might display the following error message:

  # docker push mycluster.icp:8500/open-liberty:latest
  The push refers to a repository [mycluster.icp:8500/open-liberty]
  ce633891d99e: Pushing [==================================================>]  6.656kB
  16ee2ef8c0a9: Layer already exists
  d83aa1f72c39: Layer already exists
  6b8c4250e63e: Pushing   2.56kB
  000ea2a5eb7d: Layer already exists
  5f2b8ff02676: Retrying in 5 seconds
  c1eb2e939cb4: Layer already exists
  ad60ad386e49: Retrying in 5 seconds
  db584c622b50: Retrying in 5 seconds
  52a7ea2bb533: Waiting
  52f389ea437e: Waiting
  88888b9b1b5b: Waiting
  a94e0d5a7c40: Waiting
  unknown blob

Cause

Your IBM Cloud Private cluster load balancer is configured to balance requests to master nodes. The load balancer that communicates with the registry is misdirected by the server.

Resolving the problem

If your cluster load balancer is configured, add http-request set-header X-Forwarded-Proto https if { ssl_fc } to your backend configuration. For more information, see the Docker repository on GitHub Opens in a new tab .

Scenario 2: NFS storage is not backing the storage of the registry

Symptoms

As you log on to your IBM Cloud Private cluster, you cannot push new images from your local drive because the backend storage of the Docker registry was not configured for replication. Your terminal might display the following error message:

  # docker push mycluster.icp:8500/open-liberty:latest
  The push refers to a repository [mycluster.icp:8500/open-liberty]
  ce633891d99e: Pushing [==================================================>]  6.656kB
  16ee2ef8c0a9: Layer already exists
  d83aa1f72c39: Layer already exists
  6b8c4250e63e: Pushing   2.56kB
  000ea2a5eb7d: Layer already exists
  5f2b8ff02676: Retrying in 5 seconds
  c1eb2e939cb4: Layer already exists
  ad60ad386e49: Retrying in 5 seconds
  db584c622b50: Retrying in 5 seconds
  52a7ea2bb533: Waiting
  52f389ea437e: Waiting
  88888b9b1b5b: Waiting
  a94e0d5a7c40: Waiting
  unknown blob

Cause

If your IBM Cloud Private cluster meets the following conditions, the issue is occurring because the NFS server settings do not meet the requirements for NFS of the registry:

Your IBM Cloud Private cluster is configured to use a high availability architecture.
NFS mounts for all Docker Trusted Registry (DTR) replicas are configured.

Resolving the problem

If you push an image to a high-availability DTR, be sure to configure DTR and use the required NFS server options, sync and actimeo=0. For more information, see pushing to a HA DTR from the Docker success center page Opens in a new tab .