Network observability pod issues for ContainerCreating state

This issue occurs because of a CRI-O port binding race condition during initial deployment. When you deploy the FlowCollector in direct mode, the operator creates a DaemonSet that attempts to bind hostPort 2055 on all nodes simultaneously. This can cause inconsistencies in the CRI-O port allocation table, or the OVN-Kubernetes CNI plugin that sets up network interfaces can be delayed, which causes the port to remain in pending state.

One or more of the following symptoms can occur:

• netobserv-ebpf agent pods remain in the ContainerCreating state
• flowlogs-pipeline pods remain in the ContainerCreating state
• Pod events show errors related to hostport mapping failures
• Following error messages can be displayed:

  failed to add hostport mapping for sandbox

  or

  failed to ensure nftables chains

netobserv-ebpf agent pods remain in ContainerCreating state

Complete these steps to resolve the issue:

1. Log in to the Red Hat® OpenShift® console as the admin user.
2. From the top menu, click your UserID, and click Copy login command. If prompted, log in again.
3. Click the Display token link.
4. Copy the Log in with this token command.
5. From the terminal, log in to the cluster and run the command to force delete the affected pod:

   oc delete pod netobserv-ebpf-agent-xxx -n netobserv-privileged --grace-period=0 --force

   Note: Replace netobserv-ebpf-agent-xxx with your actual pod name.

Wait for the node to be ready, then check the pod status. The pod should now be running.

flowlogs-pipeline pods remain in ContainerCreating state

Complete these steps to resolve the issues when flowlogs-pipeline agent pods remain in the ContainerCreating state due to hostport mismatch.

1. Login to the Red Hat® OpenShift®console as the admin user.
2. From the top menu, click your user ID, and click Copy login command. If prompted, log in again.
3. Click the Display token link.
4. Copy the Log in with this token command.
5. From the navigation menu, click Workloads > Pods, select the pod, and then click Events to check the events of the pod that is in the ContainerCreating state.
6. Check for errors in the event section or an error such as:

   Failed to create pod sandbox: rpc error: code = Unknown desc = failed to add hostport mapping for sandbox k8s_flowlogs-pipeline-tmtw7_netobserv_f335f380-2c6b-467b-82a6-983625d6cc38_0(f0366c3b81f9f56424979730b71c248a257b349293e98c0ad886dfc56799b733): failed to ensure nftables chains: /dev/stdin:18:43-79: Error: Could not process rule: File exists add element ip crio-hostports hostports { tcp . 2055 comment "TJN6FZKM6FQ2MVFF" : 10.254.68.116 . 2055 }

7. Identify the node where the pod is scheduled and run the following command:

   oc get pods -o wide -n netobserv

8. Debug into the node.

   oc debug node/<nodename>
   chroot /host

   Note: Replace <nodename> with your actual node name.
9. Verify the orphaned nftables rule.

   nft list ruleset | grep 2055

   Note: Replace 2055 with the port number from your error message.
10. Restart CRI-O on the node.

    systemctl restart crio

    Important: Your terminal session may disconnect after this command. This is an expected behavior.

Wait for the node to be ready, then check the pod status. The pod should now be running.