High-availability considerations

Red Hat Enterprise Linux 9.2 LPAR mode z/VM guest KVM guest

For high-availability, SMC-R setups often involve more than one network interface. When setting up such connections, you need to consider reverse-path forwarding and ARP flux.

Reverse-path forwarding

In Red Hat® Enterprise Linux®, strict reverse-path forwarding is enabled by default. With strict reverse-path forwarding, packets are forwarded only if they come in through the same interface the kernel uses to send a packet to that IP address.

Because SMC-R setups often involve more than one network interface, strict reverse-path forwarding can cause the RDMA v2 packets to be silently dropped at the destination. This prevents successfully establishing an SMC-R v2 connection. In such cases, although TCP communication between the two hosts is possible, no fallback to TCP takes place.

In Linux, reverse-path forwarding is controlled by the sysctl net.ipv4.conf.all.rp_filter. The simplest way to stop strict reverse-path forwarding from causing connection problems is to switch to loose reverse-path forwarding. To set loose reverse-path forwarding, issue:

# sysctl -w net.ipv4.conf.all.rp_filter=2

Alternatively, use /etc/sysctl.conf.

ARP flux

Prevent ARP replies on the wrong link.

Use the two sysctl commands that follow to avoid the so called ARP flux problem on Linux systems where ARP replies are sent out to the wrong network interface. This problem occurs when multiple network devices are configured for the same subnet.

# sysctl -w net.ipv4.conf.all.arp_ignore=1
# sysctl -w net.ipv4.conf.all.arp_announce=2