Configuring the network settings of hosts in a Db2 pureScale environment on an InfiniBand network (AIX)
Before you begin
- Ensure you have created your Db2 pureScale Feature installation plan. Your installation plan helps ensure that your system meets the prerequisites and that you have performed the preinstallation tasks.
- Read about supported network topologies for Db2 pureScale environments in Network topology configuration support for Db2 pureScale environments.
- Ensure that the required uDAPL for your TL level is as specified in the installation prerequisites.
- Assign InfiniBand (IB) logical adapter resources to partitions on the Hardware Management Console and start the partitions.
About this task
You must perform these steps on each host, or LPAR, you want to participate in the Db2 pureScale instance.
Cluster caching facilities (CFs) and members support multiple communication adapter ports to help Db2 pureScale environments scale and to help with high availability.
One communication adapter port for each CF or member is all that is required, though it is recommended to use more adapter ports to increase bandwidth, add redundancy, and allow the use of multiple switches. This topic guides you through the installation and setup of User Direct Access Programming Library (uDAPL) and InfiniBand on AIX® hosts and configuring IP addresses.
RestrictionsAdministrative access is required on all Db2 member and CF hosts.
- Log in as root.
- Ensure that any AIX fixes are installed from the installation prerequisites at this time.
- If file /etc/dat.conf was previously setup with the desired values, save the existing copy of dat.conf.
- Verify that your system has the correct uDAPL and InfiniBand
file sets. Run the following command, shown with sample output, to verify uDAPL is installed correctly:
The command output varies depending on version, technology level, and service pack level.
lslpp -l bos.mp64 devices.chrp.IBM.lhca.rte devices.common.IBM.ib.rte udapl.rte Fileset Level State Description ---------------------------------------------------------------------------- Path: /usr/lib/objrepos bos.mp64 22.214.171.124 APPLIED Base Operating System 64-bit Multiprocessor Runtime devices.chrp.IBM.lhca.rte 126.96.36.199 APPLIED Infiniband Logical HCA Runtime Environment devices.common.IBM.ib.rte 188.8.131.52 APPLIED Infiniband Common Runtime Environment udapl.rte 184.108.40.206 APPLIED uDAPL Path: /etc/objrepos bos.mp64 220.127.116.11 APPLIED Base Operating System 64-bit Multiprocessor Runtime devices.chrp.IBM.lhca.rte 18.104.22.168 APPLIED Infiniband Logical HCA Runtime Environment devices.common.IBM.ib.rte 22.214.171.124 APPLIED Infiniband Common Runtime Environment udapl.rte 126.96.36.199 APPLIED uDAPL ------------------------------------------------------------------------------
If any of the filesets in the previous step were newly installed
or updated, reboot the system by running the following command:
- Configure the InfiniBand subsystem and set IP addresses:
Note: In the previous example, the ib0 interface on the iba0 HCA is in the 10.222.0.0/24 subnet. To enable multiple communication adapter ports on the cluster caching facility (CF) or member, repeat steps
- Configure the InfiniBand subsystem in this substep only
if InfiniBand was never set up before on the host.
Run the smitty icm command:
- Select Add an Infiniband Communication Manager
- Key Enter and wait for the command to complete
- Exit by keying Esc+0
Infiniband Communication Manager Device Name icm Minimum Request Retries  Maximum Request Retries  Minimum Response Time (msec)  Maximum Response Time (msec)  Maximum Number of HCA's  Maximum Number of Users  Maximum Number of Work Requests  Maximum Number of Service ID's  Maximum Number of Connections  Maximum Number of Records Per Request  Maximum Queued Exception Notifications Per User  Number of MAD buffers per HCA 
- Run the smitty inet command to configure
- Select Change / Show Characteristics
- Select the first IB Interface,
- In the Change / Show an IB Interface panel set the values for
Network Interface Name,
INTERNET ADDRESS (dotted decimal),
HCA Adapter, and
Adapter's port number. Other fields can be left as the default values. The following CLI terminal screen contains the settings for an example configuration that uses two switches:
Network Interface Name ib0 INTERNET ADDRESS (dotted decimal) [10.222.0.1] Network MASK (hexadecimal or dotted decimal) [255.255.255.0] HCA Adapter [iba0] Adapter's port number  Partition Key [0xFFFF] MTU  Queue Sizes  QKey [0x1E] Superpacket off Interface Specific Network Options ('NULL' will unset the option) rfc1323  tcp_recvspace  tcp_sendspace  Current STATE up Apply change to DATABASE only no
bin step 11 for each communication adapter port on each adapter. Each communication adapter port of a host or LPAR must be on a different subnet. Repeat steps 1 - 10 on the secondary CF such that each network interface shares the subnet of the corresponding interface on the primary CF. Repeat steps 1 - 10 on each member. For each successive IB interface, the number should increase. For example, ib0 will be the first adapter and the subsequent adapter will be ib1. Each successive adapter beyond the one listed on port 0, you will need to enter smitty inet and select add an interface -> add an IB interface instead of change/show interfaces.
- Configure the InfiniBand subsystem in this substep only if InfiniBand was never set up before on the host. Run the smitty icm command:
- Reboot the systems by running the following command on
Ensure that the /etc/dat.conf file has the following format:
<interface adapter name> u2.0 nonthreadsafe default /usr/lib/libdapl/libdapl2.a(shr_64.o) IBM.1.1 "/dev/iba0 1 ib0" " "
No modifications are required unless you want to use a different InfiniBand adapter, port, or interface. If the /etc/dat.conf file was previously saved after Step 5, verify that the contents are still equivalent. If not, replace the currently dat.conf with the saved copy. For example:In the case of a CF or member, the /etc/dat.conf would resemble the following example:
- The <interface adapter name> string cannot be more than 19 characters long.
- The name within quotes ("/dev/iba0 1 ib0") is the platform-specific string. This string consists
- The device name ( /dev/iba0 ) of the communication adapter ports
- Port number ( 1 or 2 ) is the adapter's port number
- The InifiniBand adapter name ( ib0 ) for the member or CF that will run on this host.
hca0 u2.0 nonthreadsafe default /usr/lib/libdapl/libdapl2.a(shr_64.o) IBM.1.1 "/dev/iba0 1 ib0" " " hca1 u2.0 nonthreadsafe default /usr/lib/libdapl/libdapl2.a(shr_64.o) IBM.1.1 "/dev/iba0 2 ib1" " " hca2 u2.0 nonthreadsafe default /usr/lib/libdapl/libdapl2.a(shr_64.o) IBM.1.1 "/dev/iba1 1 ib2" " " hca3 u2.0 nonthreadsafe default /usr/lib/libdapl/libdapl2.a(shr_64.o) IBM.1.1 "/dev/iba1 2 ib3" " "Note: The /etc/dat.conf file must only contain entries for the adapters that are in the local host. The sample /etc/dat.conf file that is installed by default typically contains irrelevant entries. To avoid unnecessary processing of the file, make the following changes:
- Move all the Db2 pureScale cluster-related adapter entries to the top of the file.
- Comment out the irrelevant entries or remove them from the file.
- Update the /etc/hosts file
on each of the hosts so that for each host in the planned Db2
the file includes all the IP addresses of all the communication adapter ports for all
hosts in the planned environment.
The /etc/hosts file must have this format: <IP_Address> <fully_qualified_name> <short_name>. All hosts in the cluster must have the same /etc/hosts format.
For example, in a planned Db2 pureScale environment with multiple communication adapter ports on the CFs and four members, the /etc/hosts configuration file might resemble the following file:
10.222.0.1 cf1-ib0.example.com cf1-ib0 10.222.1.1 cf1-ib1.example.com cf1-ib1 10.222.2.1 cf1-ib2.example.com cf1-ib2 10.222.3.1 cf1-ib3.example.com cf1-ib3 10.222.0.2 cf2-ib0.example.com cf2-ib0 10.222.1.2 cf2-ib1.example.com cf2-ib1 10.222.2.2 cf2-ib2.example.com cf2-ib2 10.222.3.2 cf2-ib3.example.com cf2-ib3 10.222.0.101 member1-ib0.example.com member1-ib0 10.222.1.101 member1-ib1.example.com member1-ib1 10.222.0.102 member2-ib0.example.com member2-ib0 10.222.1.102 member2-ib1.example.com member2-ib1 10.222.0.103 member3-ib0.example.com member3-ib0 10.222.1.103 member3-ib1.example.com member3-ib1 10.222.0.104 member4-ib0.example.com member4-ib0 10.222.1.104 member4-ib1.example.com member4-ib1Note:
- In a four member environment that uses only one communication adapter port for each CF and member, the file would look similar to the previous example, but contain only the first IP address of each of the CFs in the previous example.
- Verify the InfiniBand subsystem. Verify the InfiniBand
components are in the Available State: For example, the system output of the following command run on a CF verifies that all four IB interfaces and adapters are available:
Use the ibstat -v command to check the InfiniBand state. Verify that the ports are active and the links are up. This check applies only for the port and interface that were previously identified in /etc/dat.conf (by default port 1 on iba0 and the ib0 interface):
# lsdev -C | grep ib ib0 Available IP over Infiniband Network Interface iba0 Available InfiniBand host channel adapter ib1 Available IP over Infiniband Network Interface iba0 Available InfiniBand host channel adapter ib2 Available IP over Infiniband Network Interface iba1 Available InfiniBand host channel adapter ib3 Available IP over Infiniband Network Interface iba1 Available InfiniBand host channel adapter icm Available Infiniband Communication Manager
Ping from each IB interfaces to every other IB interfaces in the cluster that are in the same IP subnet to make sure that they are reachable. For example,
----------------------------------------------------------------- IB PORT 1 INFORMATION (iba0) ----------------------------------------------------------------- Global ID Prefix: fe.80.00.00.00.00.00.00 Local ID (LID): 000e Local Mask Control (LMC): 0000 Logical Port State: Active Physical Port State: Active Physical Port Physical State: Link Up Physical Port Speed: 5.0G Physical Port Width: 4X Maximum Transmission Unit Capacity: 2048 Current Number of Partition Keys: 1 Partition Key List: P_Key: ffff Current Number of GUID's: 1 Globally Unique ID List: GUID: 00.02.55.00.40.61.40.00
ping -I <source IP> <destination IP>