Unable to mirror the repository using cloudctl CASE launch as a non-root user

The command cloudctl case launch --action mirror-images fails when using Skopeo as a non-root user while trying to mirror the images.

Problem

When you run the cloudctl CASE launch command (cloudctl case launch --action mirror-images) as a non-root user, the command fails with the following error:

level=fatal msg="writing signatures: mkdir /var/lib/containers/sigstore/cp: permission denied"
[INFO] Deleting mirrored image csv files created during this mirror attempt

Symptom

The cloudctl CASE launch command fails in the file /tmp/CASE.log with the following error:

    time="2022-02-15T12:54:06Z" level=debug msg="Using registries.d directory /etc/containers/registries.d for sigstore configuration"
    time="2022-02-15T12:57:37Z" level=fatal msg="writing signatures: mkdir /var/lib/containers/sigstore/cp: permission denied"
    [INFO] Deleting mirrored image csv files created during this mirror attempt

Cause

By default, the registries configuration directory is $HOME/.config/containers/registries.d, otherwise it is /etc/containers/registries.d.

If /etc/containers/registries.d exists, then the files inside that directory are processed to find out where to store the signatures. The default.yaml file inside /etc/containers/registries.d almost always refers to /var/lib/containers/. Hence, using the default.yaml file, the signature will be stored at /var/lib/containers/sigstore.

    time="2022-02-15T12:54:06Z" level=debug msg="Using registries.d directory /etc/containers/registries.d for sigstore configuration"
    time="2022-02-15T12:57:37Z" level=fatal msg="writing signatures: mkdir /var/lib/containers/sigstore/cp: permission denied"
    [INFO] Deleting mirrored image csv files created during this mirror attempt

Skopeo infers it has to write to /var/lib/containers/sigstore but that file is owned by root and cannot write to that location, causing the failure.

For more information about the Skopeo non-root behavior, see containers-registries.d.5.md.

Environment

• Mirroring Cloud Pak repository as a non-root user
• Using the cloudctl CASE command as a non-root user
• Applicable to all cloudctl versions that support Skopeo

Diagnosing the problem

There are two ways to diagnose the problem. You can choose to review the /tmp/CASE.logs or check your write access to /var/lib/containers/sigstore.

• Review the /tmp/CASE.logs to see if the following error occurs:

    msg="writing signatures: mkdir /var/lib/containers/sigstore/cp: permission denied"
    [INFO] Deleting mirrored image csv files created during this mirror attempt
  

• Run ls -l /var/lib/containers/ and see if you have no write access to /var/lib/containers/sigstore:

    ls -l /var/lib/containers/
  
    drwxr-xr-x  2 root root   6 Jan 20 01:48 sigstore
    drwx------ 10 root root 182 Jun 15  2021 storage
  

Resolving the problem

• Create the $HOME/.config/containers directory and copy the files from the /etc/containers/registries.d directory

    mkdir -p $HOME/.config/containers
    cp -r  /etc/containers/registries.d  $HOME/.config/containers/registries.d
  

• If the default.yaml file exists in the copied files, update the "sigstore" directory location. The default non-root location is file://$HOME/.local/share/containers/sigstore.

    # This is the default signature write location for docker registries.
    default-docker:
      sigstore: file:///var/lib/containers/sigstore
      sigstore-staging: file:///var/lib/containers/sigstore
  

  Update the manifests in that directory to the location where you want the signature stored.

  • If the "sigstore" location is not configured in any of the files under $HOME/.config/containers/registries.d, then the default file://$HOME/.local/share/containers/sigstore is used. You will be able to use the cloudctl CASE command when you see the following log:

       msg="Loading registries configuration \"/etc/containers/registries.conf\""
       msg="Found credentials for my-target-registry.com:5000 in credential helper containers-auth.json in file /home/airgap/.airgap/auth.json"
       msg="Using registries.d directory /home/airgap/.config/containers/registries.d for sigstore configuration"
       msg="Using \"default-docker\" configuration"
       msg="Using file:///var/lib/containers/sigstore"
       msg="Looking for TLS certificates and private keys in /etc/docker/certs.d/my-target-registry.com:5000"
       msg="Trying to access \"cp.icr.io/cp/ibm-mqadvanced-server@sha256:3248bb3ece67b71245e78b00c93b773703b08420d1276178ac7787791d8c88cc\""
       msg="Found credentials for cp.icr.io in credential helper containers-auth.json in file /home/airgap/.airgap/auth.json"
       msg="Using registries.d directory /home/airgap/.config/containers/registries.d for sigstore configuration"
       msg="Using \"default-docker\" configuration"
       msg="No signature storage configuration found for cp.icr.io/cp/ibm-mqadvanced.
                 using built-in default file:///home/airgap/.local/share/containers/sigstore"