IBM Support

CPD pods run out of disk, stuck in CrashLoopBackOff

Troubleshooting


Problem

All nodes are ready, but CPD pods are stuck in CrashLoopBackOff due to the following error:
Events:
  Type     Reason                    Age                    From               Message
  ----     ------                    ----                   ----               -------
  Normal   Scheduled                 68m                    default-scheduler  Successfully assigned cpd-operators/ibm-cpd-datarefinery-operator-84594554d9-65v8w to worker-4.ocp4-8.zenpause
  Warning  FailedCreatePodContainer  3m32s (x302 over 68m)  kubelet            unable to ensure pod container exists: failed to create container for [kubepods burstable pod54680e73-1f1b-4458-957e-9128bfc283a2] : No space left on device
And checked the disk usage of workers node as the following:
[core@worker-0 ~]$ df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs         40G     0   40G   0% /dev
tmpfs            40G   84K   40G   1% /dev/shm
tmpfs            40G   40G   39M 100% /run
tmpfs            40G     0   40G   0% /sys/fs/cgroup
/dev/sda4       300G   81G  219G  27% /sysroot
tmpfs            40G  4.0K   40G   1% /tmp
/dev/sda3       364M  197M  144M  58% /boot
overlay          40G   40G   39M 100% /etc/NetworkManager/systemConnectionsMerged
From the disk usage, it shows /run is running 100% used.

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB76","label":"Data Platform"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSHGYS","label":"IBM Cloud Pak for Data"},"ARM Category":[{"code":"a8m3p000000UoT1AAK","label":"Platform-\u003EPod"}],"ARM Case Number":"","Platform":[{"code":"PF088","label":"Red Hat OpenShift"}],"Version":"4.0.8"}]

Log InLog in to view more of this document

This document has the abstract of a technical article that is available to authorized users once you have logged on. Please use Log in button above to access the full document. After log in, if you do not have the right authorization for this document, there will be instructions on what to do next.

Document Information

Modified date:
11 July 2023

UID

ibm16832408