Troubleshooting
Problem
The Kafka broker pod iaf-system-kafka-0 enters a CrashLoopBackOff state after the Kafka PersistentVolumeClaim (PVC) reaches 100% utilization.
The broker cannot be accessed using oc rsh because the container terminates before a shell session can be established.
When the Kafka PVC is full, Kafka's KRaft metadata subsystem can fail while attempting to write a metadata snapshot. This causes the Kafka process to terminate and results in repeated pod restarts.
Symptom
The following symptoms can be observed:
oc get pod iaf-system-kafka-0 -n cp4aiopsThe pod shows:
0/1 CrashLoopBackOffwith a high or continuously increasing restart count.
Attempting to access the container fails:
oc rsh iaf-system-kafka-0 -n cp4aiopsKafka logs contain errors similar to:
ERROR ProcessTerminatingFaultHandler:
Encountered fatal fault: Unexpected error in raft IO thread
java.io.UncheckedIOException: Error writing file snapshot
Caused by: java.io.IOException: No space left on deviceThe failure can reference the KRaft metadata directory:
/var/lib/kafka/data/kafka-log0/__cluster_metadata-0/As a result, Kafka-dependent CP4AIOps processing can be impacted, including alert processing, stories, connectors, and lifecycle automation.
Document Location
Worldwide
Log InLog in to view more of this document
Was this topic helpful?
Document Information
Modified date:
21 September 2026
UID
ibm17283955