Checking InfiniBand configuration in Linux

dsh -av "ibv_devices | grep ehca" | wc –l

• If the number returned by the system matches the number of HCAs in the cluster, continue with Verifying that all HCAs are available to the logical partitions.
• If the number returned by the system does not match the number of HCAs, continue with step 3.

dsh -av "ibv_devices | grep ehca" > hca_list

A list of HCAs visible to the logical partitions plus their respective GUID will be generated.

1. From the HMC that manages the server, verify that the HCA has been assigned to the logical partition. If the HCA has not been assigned to the logical partition, see Installing or replacing an InfiniBand GX host channel adapter.

   If the device was not assigned to the logical partition, see Installing the operating system and configuring the cluster servers. After you assign the HCA to the logical partition, return to this step.

2. After you assign the HCA to the correct logical partition, run the following command:

   dsh -av "find /sys/bus/ibmebus/devices -name 'lhca*' | wc -l"

3. Select from the following options:
   • If the HCA is still not visible to the system, continue with the step 6.
   • If the HCA is visible to the system, continue with step Verifying that all HCAs are available to the logical partitions.

1. Open the Manage Serviceable Events task on the HMC that manages each server and review the error logs.
2. Fix any events that are reported against each server or HCAs in that server
3. Perform one of the following recovery procedures:
   • If all the interfaces in a logical partition are not configured, use the procedure in Recovering all of the ibx interfaces in a logical partition in the Linux operating system.
   • If only a single interface in a logical partition is not configured, use the procedure in Recovering a single ibx interface in Linux.

dsh -av "ibv_devinfo | grep PORT_ACTIVE" | wc -l

The total number of ports that are active are shown. Note that HCA has two ports.

• If the number returned by the system, divided by two, matches the number of HCAs in the cluster, continue with the procedure to verify that all HCAs are available to the logical partitions.
• If the number returned by the system, divided by two, does not match the number of HCAs, determine the inactive ports and check their cabling state by following step 3.
• If the number returned by the system, divided by two, does not match the number of HCAs and the ports are correctly connected, continue with step 5.

dsh -av "ibv_devinfo | egrep ‘hca_id|node_guid|port:|PORT_DOWN'"

• You might want to consider enabling the auto-port-detection feature of eHCA, especially if there are ports unused by purpose. In order to enable that feature add the following line to the file

  /etc/modprobe.conf.local:
  options ib_ehca nr_ports=-1

• In order get a full list of supported options run the command: modinfo ib_ehca

dsh -av "lsdev -Cc adapter | grep sn | grep -v Available"

A list of HCAs that are visible to the system but not available is shown.

dsh –av ‘netstat -i | grep 'ib.*link' | awk \'{split($4,a,"."); for
(i=5;i<=12;i++){printf a[i]}; printf "\n"}\''

[Nam] Is this send_queue_size and recv_queue_size from ipoib?

[Mark] Yes

dsh –av “ibstat –v | grep ‘tcp_send.*tcp_recv'”

Because superpackets should be on, the expected attribute value results are tcp_sendspace=524288 and tcp_recvspace=524288.

dsh –av “find /sys/class/net -name 'ib*' | xargs -I dn cat dn/mtu”

• If the MTU returned matches the expected MTU value, continue with Verifying that the network interfaces are recognized as up and available.
• If the MTU returned does not match the expected MTU value, continue with step 3.

echo <right value> > /sys/class/net/ibX/mtu