Troubleshooting

Edit online

This troubleshooting section provides guidance for diagnosing and resolving common issues encountered when running containers with a read-only root filesystem in OpenShift, including volume configuration, permission errors, and runtime connectivity problems.

Enable verbose logging

To get detailed information about volume population operations, add the following environment variable to the init container:

env:
- name: INIT_VERBOSE
  value: "true"

The output shows detailed logs with following information:

The list of directories that are being copied.
If the destination directories are already populated, it will be skipped.
If any errors are encountered during the copying process.

Common issues and solutions

All commands assume the use of the OpenShift CLI.

Issue 1: Init container fails with Permission Denied

Error: The init container fails with a Permission denied error when attempting to create the /mnt/var directory.

Cause: The PersistentVolumeClaim is not properly configured or mounted, or there are permission issues with the storage.

To fix the issue:

Verify the PVC exists and is bound:
```
oc get pvc -n <your-nam
```
The expected status is Bound.
Check if the PVC access mode matches to your storage class capabilities:
```
oc describe pvc <pvc-name> -n <your-namespace>
```

Verify the pod security context includes fsGroupChangePolicy:

securityContext:
     fsGroupChangePolicy: "OnRootMismatch"
     runAsNonRoot: true

Check the storage class permissions to ensure that the requested access mode is supported.
Review the init container volume mount configuration, and ensure that all paths are correctly specified.

Issue 2: Main container fails to start after init container success

Error: The container fails with a cannot write to /var/log: Read-only file system error when attempting to write to /var/log.

Cause: The issue occurs when the main container is missing required volume mounts or when the volume mounts are incorrectly configured.

To fix the issue:

Verify the main container has all the required volume mounts:

oc get pod <pod-name> -n <your-namespace> -o jsonpath='{.spec.containers[0].volumeMounts}'

Ensure that the volume mount paths in the main container correspond to those defined in the init container, excluding the /mnt prefix:
- Init container: mnt/var
- Main container: /var

Ensure the subPathusage is consistent between init and main container:

# Init container
   - name: config-data
     mountPath: /mnt/var
     subPath: var
   
 # Main container
   - name: config-data
     mountPath: /var
     subPath: var

Check that all the volumes defined in the pod spec are mounted in the main container .

Issue 3: Application cannot write to expected directory

Error: The container encounters a cannot create file /opt/custom/data: Read-only file system error when attempting to create the file /opt/custom/data.

Cause: This issue occurs when a custom writable directory is not included in the defined volume mounts.

To add support for a custom writable directory:

Add the custom directory to volume mounts in both init and main containers:

# Init container
volumeMounts:
- name: config-data
  mountPath: /mnt/opt/custom
  subPath: opt/custom

# Main container
volumeMounts:
- name: config-data
  mountPath: /opt/custom
  subPath: opt/custom

Add the ADDITIONAL_WRITABLE_VOLUMES environment variable to the init container:
```
# Init container only
env:
- name: ADDITIONAL_WRITABLE_VOLUMES
  value: "/opt/custom:/mnt/opt/custom"
```
For multiple custom directories, use a comma-separated list:
```
# Init container only
env:
- name: ADDITIONAL_WRITABLE_VOLUMES
  value: "/opt/custom:/mnt/opt/custom,/opt/data:/mnt/opt/data,/opt/config:/mnt/opt/config"
```
Important: Both steps are required. The volume mounts provide the necessary storage, while the environment variable instructs the init container to populate the custom directory with content from the container image. Each path pair in the comma-separated list must follow the format source:destination.

Issue 4: Init container copies files on every restart

Symptoms:

Init container takes a long time on every pod restart.
Logs show Copying /var to /mnt/var on every restart.

Cause: The issue occurs when destination directories are not recognized as already populated, or when the volume is removed between restarts.

To fix the issue:

Check init container logs to see if directories are being detected as populated:

oc logs <pod-name> -c populate-volumes -n <your-namespace>

The expected output on first run:

Copying /var to /mnt/var
   Successfully copied /var

The expected output on subsequent runs:

Skipping /var (destination /mnt/var not empty)

Ensure that the PVC is configured as a persistent volume and not as an emptyDir.
```
oc get pvc <pvc-name> -n <your-namespace>
```
Check if the volume is being deleted between restarts:
```
oc describe pvc <pvc-name> -n <your-namespace>
```
Ensure that the subPath configuration is correct, as an incorrect subPath can cause volumes to appear empty.
Ensure that the storage class supports persistent storage, not ephemeral.

Issue 5: PostgreSQL socket connection errors

Error: The application fails to connect to the server with a could not connect to server: No such file or directory error.

Is the server running locally and accepting connections on Unix domain socket /var/run/postgresql/.s.PGSQL.5432?

Cause: The issue occurs because PostgreSQL requires the /run volume to be mounted at both /run and /var/run/postgresql

To fix the issue:

Ensure that the PostgreSQL main container has both the mounts:

volumeMounts:
- name: run-volume
  mountPath: /run
- name: run-volume
  mountPath: /var/run/postgresql  # Required for socket