Verifying requirements have been met

After configuring nodes and creating the Multus public NetworkAttachmentDefinition (see Creating network attachment definitions) check that the node configurations and NetworkAttachmentDefinition configurations are compatible. To do so, verify that each node can ping pods via the public network.

Start a daemonset similar to the following example:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: multus-public-test
  namespace: openshift-storage
  labels:
    app: multus-public-test
spec:
 selector:
   matchLabels:
     app: multus-public-test
 template:
   metadata:
     labels:
       app: multus-public-test
     annotations:
       k8s.v1.cni.cncf.io/networks: openshift-storage/public-net #
   spec:
     containers:
       - name: test
         image: quay.io/ceph/ceph:v18 # image known to have ‘ping’ installed
         command:
           - sleep
           - infinity
         resources: {}

List the Multus public network IPs assigned to test pods using a command like the following example. This example command lists all IPs assigned to all test pods (each will have 2 IPs). From the output, it is easy to manually extract the IPs associated with the Multus public network.

oc -n openshift-storage describe pod -l app=multus-public-test | grep -o -E 'Add .* from .*'
Add eth0 [10.128.2.86/23] from ovn-kubernetes
Add net1 [192.168.20.22/24] from default/public-net
Add eth0 [10.129.2.173/23] from ovn-kubernetes
Add net1 [192.168.20.29/24] from default/public-net
Add eth0 [10.131.0.108/23] from ovn-kubernetes
Add net1 [192.168.20.23/24] from default/public-net

In the previous example, test pod IPs on the Multus public network are:

  • 192.168.20.22
  • 192.168.20.29
  • 192.168.20.23

Check that each node (NODE) can reach all test pod IPs over the public network:

oc debug node/NODE
Starting pod/NODE-debug ...
To use host binaries, run `chroot /host`
Pod IP: ****
If you don't see a command prompt, try pressing enter.


sh-5.1# chroot /host


sh-5.1# ping 192.168.20.22
PING 192.168.20.22 (192.168.20.22) 56(84) bytes of data.
64 bytes from 192.168.20.22: icmp_seq=1 ttl=64 time=0.093 ms
64 bytes from 192.168.20.22: icmp_seq=2 ttl=64 time=0.056 ms
^C
--- 192.168.20.22 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1046ms
rtt min/avg/max/mdev = 0.056/0.074/0.093/0.018 ms


sh-5.1# ping 192.168.20.29
PING 192.168.20.29 (192.168.20.29) 56(84) bytes of data.
64 bytes from 192.168.20.29: icmp_seq=1 ttl=64 time=0.403 ms
64 bytes from 192.168.20.29: icmp_seq=2 ttl=64 time=0.181 ms
^C
--- 192.168.20.29 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1007ms
rtt min/avg/max/mdev = 0.181/0.292/0.403/0.111 ms


sh-5.1# ping 192.168.20.23
PING 192.168.20.23 (192.168.20.23) 56(84) bytes of data.
64 bytes from 192.168.20.23: icmp_seq=1 ttl=64 time=0.329 ms
64 bytes from 192.168.20.23: icmp_seq=2 ttl=64 time=0.227 ms
^C
--- 192.168.20.23 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1047ms
rtt min/avg/max/mdev = 0.227/0.278/0.329/0.051 ms

If any node does not get a successful ping to a running pod, it is not safe to proceed. Diagnose and fix the issue, then repeat this testing. Some reasons you may encounter a problem include:

  • The host may not be properly attached to the Multus public network (via Macvlan).
  • The host may not be properly configured to route to the pod IP range.
  • The public NetworkAttachmentDefinition may not be properly configured to route back to the host IP range.
  • The host may have a firewall rule blocking the connection in either direction.
  • The network switch may have a firewall or security rule blocking the connection.

Suggested debugging steps:

  • Ensure nodes can ping each other over using public network “shim” IPs.
  • Ensure the output of ip address.