Installing Linux

Use this procedure if you are installing the Linux® operating system, and you were directed to it from another procedure.

Perform the installation instructions for the Linux operating system. For details, see the documentation provided with SUSE Linux Enterprise Server 10 (SP2), with IBM® InfiniBand GX host channel adapter (HCA) driver, and Open Fabrics Enterprise Distribution (OFED) stack. Alternatively, see the instructions contained in the OFED-1.3.1 package, which is available from the Open Fabrics Alliance Web site Downloads page. For more information, see the IBM Clusters with the InfiniBand Switch Web site.
Confirm that ofed, ofed-kmp-ppc64, and ofed-kmp-kdump RPM package managers (rpms) are installed. Also, check for the optional ofed-doc rpm. Use the rpm command as shown in the following example:
```
[root on c697f1sq01][/etc/sysconfig/network] => rpm -qa | grep -i ofed
ofed-kmp-kdump-1.3_2.6.16.60_0.25-0.28
ofed-1.3-0.28
ofed-doc-1.3-0.28
ofed-kmp-ppc64-1.3_2.6.16.60_0.25-0.28
```
If the above rpms have not been installed, install them now. Use the instructions from the documentation that is provided with SUSE Linux Enterprise Server 10 (SP2), with IBM InfiniBand GX HCA driver, and Open Fabrics Enterprise Distribution (OFED) stack. Alternatively, see the instructions contained in OFED-1.3.1 package available from the Open Fabrics Alliance Web site. For more information, see the IBM Clusters with the InfiniBand Switch Web site.

Set up configuration files:

Edit the /etc/modprobe.conf.local file:

If all InfiniBand HCA ports are to be used consecutively, use the nr_ports=-1 option. Otherwise, go to step 4.a.ii.

#
# add local extensions to this file
#
options ib_ehca nr_ports=-1
options ib_ipoib send_queue_size=256 recv_queue_size=512

If only the first port in each InfiniBand HCA is to be used, use the nr_ports=1 option:

#
# add local extensions to this file
#
options ib_ehca nr_ports=1
options ib_ipoib send_queue_size=256 recv_queue_size=512

Edit the /etc/sysconfig/network/ifcfg-ibX file, where X is the InfiniBand interface number. There is one file for each active interface. These files are read at boot time when the IPoIB code is loaded.

The following are the important fields.

BOOTPROTO=
BROADCAST=
IPADDR=  # ip address of the interface
MTU=  # MTU for the interface (see Planning for maximum transfer units (MTUs) 
and the QLogic switch planning work sheets. 
A blank value = 2KB MTU default)
NETMASK=
NETWORK=
REMOTE_IPADDR=
STARTMODE=

Example: A server with four InfiniBand interfaces could have files similar to the following example.

[root on c697f1sq01][/etc/sysconfig/network] => cat ifcfg-ib0
BOOTPROTO='static'
BROADCAST='10.0.1.255'
IPADDR='10.0.1.1'
MTU=''
NETMASK='255.255.255.0'
NETWORK='10.0.1.0'
REMOTE_IPADDR=''
STARTMODE='onboot'

[root on c697f1sq01][/etc/sysconfig/network] => cat ifcfg-ib1
BOOTPROTO='static'
BROADCAST='10.0.2.255'
IPADDR='10.0.2.1'
MTU=''
NETMASK='255.255.255.0'
NETWORK='10.0.2.0'
REMOTE_IPADDR=''
STARTMODE='onboot'

[root on c697f1sq01][/etc/sysconfig/network] => cat ifcfg-ib2
BOOTPROTO='static'
BROADCAST='10.0.3.255'
IPADDR='10.0.3.1'
MTU=''
NETMASK='255.255.255.0'
NETWORK='10.0.3.0'
REMOTE_IPADDR=''
STARTMODE='onboot'

[root on c697f1sq01][/etc/sysconfig/network] => cat ifcfg-ib3
BOOTPROTO='static'
BROADCAST='10.0.4.255'
IPADDR='10.0.4.1'
MTU=''
NETMASK='255.255.255.0'
NETWORK='10.0.4.0'
REMOTE_IPADDR=''
STARTMODE='onboot'

Restart the server after you set up configuration files.
Verify that the IPoIB process starts. Use the lsmod command.

Use the ifconfig ibX command to verify interface operation.

Example output (note the correct configuration based on the /etc/sysconfig/ifcfg-ib0 file, and that the broadcast is running.):

[root on c697f1sq01][/etc/sysconfig/network] => ifconfig ib0
ib0       Link encap:UNSPEC  HWaddr 80-00-08-24-FE-80-00-00-00-00-00-00-00-00-00-00
          inet addr:10.0.1.1  Bcast:10.0.1.255  Mask:255.255.255.0
          inet6 addr: fe80::202:5500:1001:2900/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:2044  Metric:1
          RX packets:895100 errors:0 dropped:0 overruns:0 frame:0
          TX packets:89686 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:512
          RX bytes:50136680 (47.8 Mb)  TX bytes:5393192 (5.1 Mb)

Use the netstat –i command to verify the interface table.

Example output with 4 KB MTU configuration:

Iface   MTU Met   RX-OKB RX-ERR RX-DRP RX-OVR   TX-OKB TX-ERR TX-DRP TX-OVR Flg
eth0   1500   0 1141647      0      0      0  122790      0      0      0 BMRU
ib0    4092   0 1028150      0      0      0  102996      0      0      0 BMRU
ib1    4092   0 1028260      0      0      0  102937      0      0      0 BMRU
ib2    4092   0 1028494      0      0      0  102901      0      0      0 BMRU
ib3    4092   0 1028293      0      0      0  102910      0      0      0 BMRU
lo    16436   0  513906      0      0      0  513906      0      0      0 LRU

Use the netstat –rn command to verify the routing table.

Example output:

[root on c697f1sq01][/etc/init.d] => netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
9.114.28.64     0.0.0.0         255.255.255.192 U         0 0          0 eth0
10.0.4.0        0.0.0.0         255.255.255.0   U         0 0          0 ib3
10.0.1.0        0.0.0.0         255.255.255.0   U         0 0          0 ib0
10.0.2.0        0.0.0.0         255.255.255.0   U         0 0          0 ib1
10.0.3.0        0.0.0.0         255.255.255.0   U         0 0          0 ib2
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 eth0
127.0.0.0       0.0.0.0         255.0.0.0       U         0 0          0 lo
0.0.0.0         9.114.28.126    0.0.0.0         UG        0 0          0 eth0

Return to the procedure that directed you here.