Unable to mirror the repository using cloudctl CASE launch as a non-root user

Edit online

The command cloudctl case launch --action mirror-images fails when using Skopeo as a non-root user while trying to mirror the images.

Problem

When you run the cloudctl CASE launch command (cloudctl case launch --action mirror-images) as a non-root user, the command fails with the following error:

level=fatal msg="writing signatures: mkdir /var/lib/containers/sigstore/cp: permission denied"
[INFO] Deleting mirrored image csv files created during this mirror attempt

Symptom

The cloudctl CASE launch command fails in the file /tmp/CASE.log with the following error:

time="2022-02-15T12:54:06Z" level=debug msg="Using registries.d directory /etc/containers/registries.d for sigstore configuration"
   time="2022-02-15T12:57:37Z" level=fatal msg="writing signatures: mkdir /var/lib/containers/sigstore/cp: permission denied"
   [INFO] Deleting mirrored image csv files created during this mirror attempt

Cause

By default, the registries configuration directory is $HOME/.config/containers/registries.d, otherwise it is /etc/containers/registries.d.

If /etc/containers/registries.d exists, then the files inside that directory are processed to find out where to store the signatures. The default.yaml file inside /etc/containers/registries.d almost always refers to /var/lib/containers/. Hence, using the default.yamlfile, the signature will be stored at /var/lib/containers/sigstore.

time="2022-02-15T12:54:06Z" level=debug msg="Using registries.d directory /etc/containers/registries.d for sigstore configuration"
   time="2022-02-15T12:57:37Z" level=fatal msg="writing signatures: mkdir /var/lib/containers/sigstore/cp: permission denied"
   [INFO] Deleting mirrored image csv files created during this mirror attempt

Skopeo infers it has to write to /var/lib/containers/sigstore but that file is owned by root and cannot write to that location, causing the failure.

For more information about the Skopeo non-root behavior, see containers-registries.d.5.md.

Environment

Mirroring Cloud Pak repository as a non-root user
Using the cloudctl CASE command as a non-root user
Applicable to all cloudctl versions that support Skopeo

Diagnosing the problem

There are two ways to diagnose the problem. You can choose to review the /tmp/CASE.logs or check your write access to /var/lib/containers/sigstore.

Review the /tmp/CASE.logs to see if the following error occurs:

msg="writing signatures: mkdir /var/lib/containers/sigstore/cp: permission denied"
 [INFO] Deleting mirrored image csv files created during this mirror attempt

Run ls -l /var/lib/containers/ and see if you have no write access to /var/lib/containers/sigstore:

ls -l /var/lib/containers/

 drwxr-xr-x  2 root root   6 Jan 20 01:48 sigstore
 drwx------ 10 root root 182 Jun 15  2021 storage

Resolving the problem

Create the $HOME/.config/containers directory and copy the files from the /etc/containers/registries.d directory

mkdir -p $HOME/.config/containers
 cp -r  /etc/containers/registries.d  $HOME/.config/containers/registries.d

If the default.yaml file exists in the copied files, update the "sigstore" directory location. The default non-root location is file://$HOME/.local/share/containers/sigstore.
```
# This is the default signature write location for docker registries.
 default-docker:
   sigstore: file:///var/lib/containers/sigstore
   sigstore-staging: file:///var/lib/containers/sigstore
```
Update the manifests in that directory to the location where you want the signature stored.

If the "sigstore" location is not configured in any of the files under $HOME/.config/containers/registries.d, then the default file://$HOME/.local/share/containers/sigstore is used. You will be able to use the cloudctl CASE command when you see the following log:

msg="Loading registries configuration \"/etc/containers/registries.conf\""
 msg="Found credentials for my-target-registry.com:5000 in credential helper containers-auth.json in file /home/airgap/.airgap/auth.json"
 msg="Using registries.d directory /home/airgap/.config/containers/registries.d for sigstore configuration"
 msg="Using \"default-docker\" configuration"
 msg="Using file:///var/lib/containers/sigstore"
 msg="Looking for TLS certificates and private keys in /etc/docker/certs.d/my-target-registry.com:5000"
 msg="Trying to access \"cp.icr.io/cp/ibm-mqadvanced-server@sha256:3248bb3ece67b71245e78b00c93b773703b08420d1276178ac7787791d8c88cc\""
 msg="Found credentials for cp.icr.io in credential helper containers-auth.json in file /home/airgap/.airgap/auth.json"
 msg="Using registries.d directory /home/airgap/.config/containers/registries.d for sigstore configuration"
 msg="Using \"default-docker\" configuration"
 msg="No signature storage configuration found for cp.icr.io/cp/ibm-mqadvanced.
           using built-in default file:///home/airgap/.local/share/containers/sigstore"