Moving from a TCP/IP protocol network to an RDMA protocol network

In Db2® Cancun Release 10.5.0.4 and later fix packs, you can run a Db2 pureScale® environment on a Transmission Control Protocol/Internet Protocol over Ethernet (TCP/IP) network. A TCP/IP protocol network does not require special remote direct memory access (RDMA) capable adapters.

Before you begin

Important: Starting from version 11.5.5, support for Infiniband (IB) adapters as the high-speed communication network between members and CFs in Db2 pureScale on all supported platforms is deprecated and will be removed in a future release. Use Remote Direct Memory Access over Converged Ethernet (RoCE) network as the replacement.

Ensure that you created your Db2 pureScale Feature installation plan. Your installation plan helps ensure that your system meets the prerequisites and that you perform the preinstallation tasks that are required for an InfiniBand network or a RoCE network. (Such as, installing the required RDMA software, updating the /etc/hosts file on each of host, and details regarding the communication adapter ports and switches.)

About this task

Running your Db2 pureScale environment on a TCP/IP network can provide a faster setup for testing the technology. However, for the most demanding write-intensive data sharing workloads, an RDMA protocol over Converged Ethernet (RoCE) network can offer better performance.

Attention: Recent conversions of customer clusters from TCP/IP have identified two parameters that need to be set for a smooth transition from TCP/IP to RDMA technologies:

Verify that all RoCE adapters have the MTU set to 1500. Using non-standard maximum transmission units (MTUs) can lead to RDMA packet loss for larger packets and can cause communication failures at database start or activation. For more information, see Installation prerequisites for the Db2 pureScale environment.
Set ipcmni_extend to allow SEMMNI to exceed 32768. For hosts with more than 128Gb RAM, set the kernel boot parameter ipcmni_extend to allow the kernel parameter SEMMNI to exceed 32768. If this parameter is not set, the instance or database can fail to allocate memory or IPC resources and might fail to start. For more information, see Kernel parameter requirements.

Procedure

To move your existing Db2 pureScale instance from a TCP/IP protocol network to an RDMA protocol network:

Configure the communication adapter ports and switches for the RDMA protocol network (as detailed in the appropriate AIX® or Linux® topics).
Ensure that you check the RDMA requirements. On a Linux operating system, you must also validate the prerequisites for your specific type of network configuration. (The network configuration type can be a single InfiniBand port cluster, multiple InfiniBand port cluster, single RoCE port cluster, or multiple RoCE port cluster.) To check the requirements, run the db2prereqcheck command.
For example, check the RDMA requirements on a multi port RoCE cluster network:
```
db2prereqcheck -hl <hostA> -nm <netnameA1, netnameA2> -hl <hostB> -nm <netnameB1, netnameB2>
```
Stop the Db2 pureScale instance (db2stop command). Ensure all members and CFs are stopped.
If you are using the same adapters for the RDMA network, skip to Step 6. Otherwise, if you are using different adapters, you must update the netnames on the members and the CFs with the db2iupdt -update command.
For example:
1. To update the member netnames:
```
db2iupdt -update -m MemberHostName -mnet MemberNetName,MemberNetName,MemberNetName -u FencedID InstName
```
2. To update the CF netnames:
```
db2iupdt -update -cf CFHostName -cfnet CFNetName,CFNetName -u FencedID InstName
```

Verify the db2nodes.cfg file is updated and indicates an RDMA protocol.

For example:

0 coralxib10 0 coralxib10-en1 - MEMBER
1 coralxib11 0 coralxib11-en1 - MEMBER
2 coralxib12 0 coralxib12-en1 - MEMBER
128 coralxib13 0 coralxib13-en1 - CF
129 coralxib14 0 coralxib14-en1 - CF

Update the database manager configuration parameter CF_TRANSPORT_METHOD to RDMA.

For example:

db2 update dbm cfg using CF_TRANSPORT_METHOD RDMA

Verify the transport method is RDMA. For example:

db2 get dbm cfg | grep -i transp

Transport method to CF          (CF_TRANSPORT_METHOD) = RDMA

Start the Db2 pureScale instance (db2start command).

Results

Your Db2 pureScale environment is now running on an RDMA protocol network.