IBM Support

Kafka CrashLoopBackOff Due to Full Persistent Volume and KRaft Metadata Snapshot Failure

Troubleshooting


Problem

The Kafka broker pod iaf-system-kafka-0 enters a CrashLoopBackOff state after the Kafka PersistentVolumeClaim (PVC) reaches 100% utilization.

The broker cannot be accessed using oc rsh because the container terminates before a shell session can be established.

When the Kafka PVC is full, Kafka's KRaft metadata subsystem can fail while attempting to write a metadata snapshot. This causes the Kafka process to terminate and results in repeated pod restarts.

Symptom

The following symptoms can be observed:

oc get pod iaf-system-kafka-0 -n cp4aiops

The pod shows:

0/1   CrashLoopBackOff

with a high or continuously increasing restart count.

Attempting to access the container fails:

oc rsh iaf-system-kafka-0 -n cp4aiops

Kafka logs contain errors similar to:

ERROR ProcessTerminatingFaultHandler:
Encountered fatal fault: Unexpected error in raft IO thread

java.io.UncheckedIOException: Error writing file snapshot

Caused by: java.io.IOException: No space left on device

The failure can reference the KRaft metadata directory:

/var/lib/kafka/data/kafka-log0/__cluster_metadata-0/

As a result, Kafka-dependent CP4AIOps processing can be impacted, including alert processing, stories, connectors, and lifecycle automation.

 

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB77","label":"Automation Platform"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSE9G0Q","label":"IBM Cloud Pak for AIOps"},"ARM Category":[{"code":"a8m3p000000hAHpAAM","label":"Cloud Pak for AIOps-\u003EAI Manager"}],"ARM Case Number":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"4.13.1"}]

Log InLog in to view more of this document

This document has the abstract of a technical article that is available to authorized users once you have logged on. Please use Log in button above to access the full document. After log in, if you do not have the right authorization for this document, there will be instructions on what to do next.

Document Information

Modified date:
21 September 2026

UID

ibm17283955