IBM Support

API Connect OVA: Excessive portal-sync-backup Pods in Pending Status Impacting Healthcheck

Troubleshooting


Problem

In API Connect OVA deployments, customers may observe hundreds of *-sync-backup pods in a Pending state. These pods are generated by the *-sync-backup-schedule job, which runs every ten minutes. Over time, this results in a large number of pending pods accumulating on the portal node.
This condition can lead to failures when running the apic healthcheck command from within the VM shell, due to resource exhaustion or container runtime instability.

Diagnosing The Problem

  1. Access the Portal Node 
    ssh to the portal node
apicadm@subinvm1:~# sudo -i
  2. Check for sync-backup jobs and pods 
    kubectl get job -o name | grep sync-backup
kubectl get pods | grep sync-backup
  3. Count the number of sync-backup jobs 
    kubectl get job -o name | grep sync-backup | wc -l
  4. Run the healthcheck< 
    apic healthcheck
If the command fails, note any errors related to pod status or resource limits.

Resolving The Problem

Please ensure you have proper backups. For more information about backups click here

Step 1: Remove Excess Sync-Backup Jobs

for name in $(kubectl get job -o name | grep sync-backup); do
  kubectl delete $name
done

Verify that the jobs have been removed:

kubectl get job -o name | grep sync-backup | wc -l

Check if the pending pods have cleared:

kubectl get pods | grep sync-backup

Wait approximately 5 minutes to allow Kubernetes to clean up the pods.

Step 2: Restart Container Runtime (if pods persist)

If pending pods remain after job deletion, the container runtime (containerd) may be in a degraded state. To resolve :

systemctl status containerd
systemctl restart containerd
systemctl status containerd
kubectl get pods | grep sync-backup

Step 3: Check for Resource Exhaustion

If the issue persists and the following command fails:

crictl pods

With an error similar to:

code = ResourceExhausted desc = grpc: trying to send message larger than max (16835417 vs. 16777216)

Refer to Technote click here for instructions on resolving containerd gRPC message size limits.

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB77","label":"Automation Platform"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSMNED","label":"IBM API Connect"},"ARM Category":[{"code":"a8mKe000000CaZXIA0","label":"API Connect-\u003EAPIC Platform - Other"}],"ARM Case Number":"TS020173078","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"10.0.5"}]

Document Information

Modified date:
11 September 2025

UID

ibm17244679

Page Feedback