Configuring the network settings of hosts in a Db2 pureScale environment on an InfiniBand network (AIX)

As described in the network topology tables and diagrams, configure the communication adapter ports in pairs, so that the devices with the same device ID (for example, ib0) are on the same subnet.

Before you begin

Important: Starting from version 11.5.5, support for Infiniband (IB) adapters as the high-speed communication network between members and CFs in Db2 pureScale on all supported platforms is deprecated and will be removed in a future release. Use Remote Direct Memory Access over Converged Ethernet (RoCE) network as the replacement.

Ensure that you complete the following tasks:

Ensure you have created your Db2 pureScale Feature installation plan. Your installation plan helps ensure that your system meets the prerequisites and that you have performed the preinstallation tasks.
Read about supported network topologies for Db2 pureScale environments in Network topology configuration support for Db2 pureScale environments.
Ensure that the required uDAPL for your TL level is as specified in the installation prerequisites.
Assign InfiniBand (IB) logical adapter resources to partitions on the Hardware Management Console and start the partitions.

About this task

You must perform these steps on each host, or LPAR, you want to participate in the Db2 pureScale instance.

Cluster caching facilities (CFs) and members support multiple communication adapter ports to help Db2 pureScale environments scale and to help with high availability.

One communication adapter port for each CF or member is all that is required, though it is recommended to use more adapter ports to increase bandwidth, add redundancy, and allow the use of multiple switches. This topic guides you through the installation and setup of User Direct Access Programming Library (uDAPL) and InfiniBand on AIX® hosts and configuring IP addresses.

Restrictions

Administrative access is required on all Db2 member and CF hosts.

Procedure

Log in as root.
Ensure that any AIX fixes are installed from the installation prerequisites at this time.
If file /etc/dat.conf was previously setup with the desired values, save the existing copy of dat.conf.

Verify that your system has the correct uDAPL and InfiniBand file sets.

Run the following command, shown with sample output, to verify uDAPL is installed correctly:

lslpp -l bos.mp64 devices.chrp.IBM.lhca.rte devices.common.IBM.ib.rte udapl.rte

  Fileset                      Level  State      Description
  ----------------------------------------------------------------------------
Path: /usr/lib/objrepos
  bos.mp64                   7.1.5.32  APPLIED    Base Operating System 64-bit
                                                  Multiprocessor Runtime
  devices.chrp.IBM.lhca.rte  7.1.5.30  APPLIED    Infiniband Logical HCA Runtime
                                                  Environment
  devices.common.IBM.ib.rte  7.1.5.30  APPLIED    Infiniband Common Runtime
                                                  Environment
  udapl.rte                  7.1.5.0   COMMITTED  uDAPL

Path: /etc/objrepos
  bos.mp64                   7.1.5.32  APPLIED    Base Operating System 64-bit
                                                  Multiprocessor Runtime
  devices.chrp.IBM.lhca.rte  7.1.3.15  COMMITTED  Infiniband Logical HCA Runtime
                                                  Environment
  devices.common.IBM.ib.rte  7.1.5.30  APPLIED    Infiniband Common Runtime
                                                  Environment
  udapl.rte                  7.1.5.0   COMMITTED  uDAPL

------------------------------------------------------------------------------

The command output varies depending on version, technology level, and service pack level.

If any of the filesets in the previous step (step 4) were newly installed or updated, reboot the system by running the following command:
```
 shutdown -Fr
```

Configure the InfiniBand subsystem and set IP addresses:

Configure the InfiniBand subsystem in this substep only if InfiniBand was never set up before on the host. Run the smitty icm command:

Select Add an Infiniband Communication Manager
Key Enter and wait for the command to complete
Exit by keying Esc+0

For example,

Infiniband Communication Manager Device Name        icm
Minimum Request Retries                            [1]
Maximum Request Retries                            [7]
Minimum Response Time (msec)                       [100]
Maximum Response Time (msec)                       [4300]
Maximum Number of HCA's                            [256]
Maximum Number of Users                            [65000]
Maximum Number of Work Requests                    [65000]
Maximum Number of Service ID's                     [1000]
Maximum Number of Connections                      [65000]
Maximum Number of Records Per Request              [64]
Maximum Queued Exception Notifications Per User    [1000]
Number of MAD buffers per HCA                      [64]

Run the smitty inet command to configure IP addresses.

Select Change / Show Characteristics
Select the first IB Interface, ib0.
In the Change / Show an IB Interface panel set the values for the fields Network Interface Name, INTERNET ADDRESS (dotted decimal), Network Mask, HCA Adapter, and Adapter's port number. Other fields can be left as the default values. The following CLI terminal screen contains the settings for an example configuration that uses two switches:

Network Interface Name                              ib0
INTERNET ADDRESS (dotted decimal)                  [10.222.0.1]
Network MASK (hexadecimal or dotted decimal)       [255.255.255.0]
HCA Adapter                                        [iba0]
Adapter's port number                              [1]
Partition Key                                      [0xFFFF]
MTU                                                [65532]
Queue Sizes                                        [4000]
QKey                                               [0x1E]
Superpacket                                         off
Interface Specific Network Options
   ('NULL' will unset the option)
  rfc1323                                          [1]
  tcp_recvspace                                    [262144]
  tcp_sendspace                                    [262144]
Current STATE                                       up
Apply change to DATABASE only                       no

Note: In the previous example, the ib0 interface on the iba0 HCA is in the 10.222.0.0/24 subnet. To enable multiple communication adapter ports on the cluster caching facility (CF) or member, repeat steps i - iii of Step 6, part b for each communication adapter port on each adapter. Each communication adapter port of a host or LPAR must be on a different subnet. Repeat steps 1 - 10 on the secondary CF such that each network interface shares the subnet of the corresponding interface on the primary CF. Repeat step 1 - step 10 on each member. For each successive IB interface, the number should increase. For example, ib0 will be the first adapter and the subsequent adapter will be ib1. For each successive adapter beyond the one listed on port 0, you will need to enter smitty inet and select add an interface -> add an IB interface instead of change/show interfaces.

Reboot the systems by running the following command on each host:
```
 shutdown -Fr
```
Ensure that the /etc/dat.conf file has the following format:
```
<interface adapter name> u2.0 nonthreadsafe default /usr/lib/libdapl/libdapl2.a(shr_64.o) IBM.1.1 "/dev/iba0 1 ib0" " "
```
- The <interface adapter name> string cannot be more than 19 characters long.
- The name within quotes ("/dev/iba0 1 ib0") is the platform-specific string. This string consists of:
  - The device name ( /dev/iba0 ) of the communication adapter ports
  - Port number ( 1 or 2 ) is the adapter's port number
  - The InifiniBand adapter name ( ib0 ) for the member or CF that will run on this host.
No modifications are required unless you want to use a different InfiniBand adapter, port, or interface. If the /etc/dat.conf file was previously saved after Step 5, verify that the contents are still equivalent. If not, replace the currently dat.conf with the saved copy. For example:
In the case of a CF or member, the /etc/dat.conf would resemble the following example:
```
hca0 u2.0 nonthreadsafe default /usr/lib/libdapl/libdapl2.a(shr_64.o) IBM.1.1 
		"/dev/iba0 1 ib0" " "
hca1 u2.0 nonthreadsafe default /usr/lib/libdapl/libdapl2.a(shr_64.o) IBM.1.1 
		"/dev/iba0 2 ib1" " "
hca2 u2.0 nonthreadsafe default /usr/lib/libdapl/libdapl2.a(shr_64.o) IBM.1.1 
		"/dev/iba1 1 ib2" " "
hca3 u2.0 nonthreadsafe default /usr/lib/libdapl/libdapl2.a(shr_64.o) IBM.1.1 
		"/dev/iba1 2 ib3" " "
```
Note: The /etc/dat.conf file must only contain entries for the adapters that are in the local host. The sample /etc/dat.conf file that is installed by default typically contains irrelevant entries. To avoid unnecessary processing of the file, make the following changes:
- Move all the Db2 pureScale cluster-related adapter entries to the top of the file.
- Comment out the irrelevant entries or remove them from the file.

Update the /etc/hosts file on each of the hosts so that for each host in the planned Db2 pureScale environment, the file includes all the IP addresses of all the communication adapter ports for all hosts in the planned environment.

The /etc/hosts file must have this format: <IP_Address> <fully_qualified_name> <short_name>. All hosts in the cluster must have the same /etc/hosts format.

For example, in a planned Db2 pureScale environment with multiple communication adapter ports on the CFs and four members, the /etc/hosts configuration file might resemble the following file:

 10.222.0.1       cf1-ib0.example.com cf1-ib0
 10.222.1.1       cf1-ib1.example.com cf1-ib1
 10.222.2.1       cf1-ib2.example.com cf1-ib2
 10.222.3.1       cf1-ib3.example.com cf1-ib3
 10.222.0.2       cf2-ib0.example.com cf2-ib0
 10.222.1.2       cf2-ib1.example.com cf2-ib1
 10.222.2.2       cf2-ib2.example.com cf2-ib2
 10.222.3.2       cf2-ib3.example.com cf2-ib3
 10.222.0.101     member1-ib0.example.com member1-ib0
 10.222.1.101     member1-ib1.example.com member1-ib1
 10.222.0.102     member2-ib0.example.com member2-ib0
 10.222.1.102     member2-ib1.example.com member2-ib1
 10.222.0.103     member3-ib0.example.com member3-ib0
 10.222.1.103     member3-ib1.example.com member3-ib1
 10.222.0.104     member4-ib0.example.com member4-ib0
 10.222.1.104     member4-ib1.example.com member4-ib1

Note:

In a four member environment that uses only one communication adapter port for each CF and member, the file would look similar to the previous example, but contain only the first IP address of each of the CFs in the previous example.

Verify the InfiniBand subsystem. Verify the InfiniBand components are in the Available State:

For example, the system output of the following command run on a CF verifies that all four IB interfaces and adapters are available:

 # lsdev -C | grep ib
 ib0        Available               IP over Infiniband Network Interface
 iba0       Available               InfiniBand host channel adapter
 ib1        Available               IP over Infiniband Network Interface
 iba0       Available               InfiniBand host channel adapter
 ib2        Available               IP over Infiniband Network Interface
 iba1       Available               InfiniBand host channel adapter
 ib3        Available               IP over Infiniband Network Interface
 iba1       Available               InfiniBand host channel adapter
 icm        Available               Infiniband Communication Manager

Use the ibstat -v command to check the InfiniBand state. Verify that the ports are active and the links are up. This check applies only for the port and interface that were previously identified in /etc/dat.conf (by default port 1 on iba0 and the ib0 interface):

 -----------------------------------------------------------------
  IB PORT 1 INFORMATION (iba0)
 -----------------------------------------------------------------
 Global ID Prefix:                       fe.80.00.00.00.00.00.00
 Local ID (LID):                         000e
 Local Mask Control (LMC):               0000
 Logical Port State:                     Active
 Physical Port State:                    Active
 Physical Port Physical State:           Link Up
 Physical Port Speed:                    5.0G
 Physical Port Width:                    4X
 Maximum Transmission Unit Capacity:     2048
 Current Number of Partition Keys:       1
 Partition Key List:
   P_Key[0]:                             ffff
 Current Number of GUID's:               1
 Globally Unique ID List:
   GUID[0]:                              00.02.55.00.40.61.40.00

Ping from each IB interfaces to every other IB interfaces in the cluster that are in the same IP subnet to make sure that they are reachable. For example,

ping -I <source IP> <destination IP>

What to do next

Configure the storage for the Db2 pureScale environment.