Troubleshooting
Problem
When upgrading to version 11.7.1.4, IBM Information Server Microservices tier installer migrates all Kubernetes cluster nodes from Docker to containerd runtime. For a multi-node environment, this involves resetting and rejoining workers. It has been observed that the worker reset may fail to clean up cni0 network interface on a random basis, leading to further problems in the upgrade flow.
Symptom
Multi-replica services, such as Kafka, Solr or Zookeeper, may show pods stuck in ContainerCreating state. When introspecting such pods with "kubectl describe pod POD_NAME", one can see the following or similar error message:
Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "4ae1e49fc763f2062e064cb26cc71607cbc7de2dbdc1b627eb6c57176a7245b1": plugin type="flannel" failed (add): failed to delegate add: failed to set bridge addr: "cni0" already has an IP address different from 10.32.1.1/24
Furthermore, the upgrade may fail with the following or similar error:
fatal: [deployment_coordinator]: FAILED! => {"attempts": 150, "changed": false, "cmd": ["kubectl", "rollout", "status", "statefulset", "solr", "-w=false", "-n", "default"], "delta": "0:00:00.155591", "end": "2022-09-26 10:58:04.739568", "rc": 0, "start": "2022-09-26 10:58:04.583977", "stderr": "", "stderr_lines": [], "stdout": "Waiting for 2 pods to be ready...", "stdout_lines": ["Waiting for 2 pods to be ready..."]}
Document Location
Worldwide
[{"Type":"MASTER","Line of Business":{"code":"LOB76","label":"Data Platform"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSZJPZ","label":"IBM InfoSphere Information Server"},"ARM Category":[{"code":"a8m0z0000001i9oAAA","label":"Microservices Tier and Kubernetes Issues"}],"ARM Case Number":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"11.7.1"}]
Log InLog in to view more of this document
This document has the abstract of a technical article that is available to authorized users once you have logged on. Please use Log in button above to access the full document. After log in, if you do not have the right authorization for this document, there will be instructions on what to do next.
Was this topic helpful?
Document Information
Modified date:
25 April 2025
UID
ibm16824799