Verifying RDMA and connectivity

Verify that RDMA is enabled and that client-server communication works.

Before you begin

Before you begin, make sure that you have the following prerequisites in place:

Ensure RDMA adapters are properly configured on both the client and the server.

RDMA stack and drivers must be loaded successfully.

[root@host01:~]# rdma link show
link bnxt_re0/1 state ACTIVE physical_state LINK_UP netdev eno12399np0

Verify connectivity between RDMA interfaces

Procedure

For NFS Server, run the ib_write_bw command.

For example,

[root@host01:~]# ib_write_bw

**************************************
* Waiting for Client to connect...   *
**************************************
---------------------------------------------------------------------------------------
                    RDMA_Write BW Test
Dual-port       : OFF           Device         : bnxt_re0
Number of qps   : 1             Transport type : IB
Connection type : RC            Using SRQ      : OFF
PCIe relax order: ON            Lock-free      : OFF
ibv_wr* API     : OFF           Using DDP      : OFF
CQ Moderation   : 1
CQE Poll Batch  : 16
Mtu             : 1024[B]
Link type       : Ethernet
GID index       : 3
Max inline data : 0[B]
rdma_cm QPs     : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
Local address:  LID 0000 QPN 0x0084 PSN 0xb35221 RKey 0x0000c01 VAddr 0x007f6376c67000
GID: 00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:255:255:10:64:04:11:15

For NFS Client, run the ib_write_bw command.

[root@host01:~]# ib_write_bw 10.64.4.115

---------------------------------------------------------------------------------------
                    RDMA_Write BW Test
Dual-port       : OFF           Device         : bnxt_re0
Number of qps   : 1             Transport type : IB
Connection type : RC            Using SRQ      : OFF
PCIe relax order: ON            Lock-free      : OFF
ibv_wr* API     : OFF           Using DDP      : OFF
Tx depth        : 128
CQ Moderation   : 1
CQE Poll Batch  : 16
Mtu             : 1024[B]
Link type       : Ethernet
GID index       : 3
Max inline data : 0[B]
rdma_cm QPs     : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address:  LID 0000 QPN 0x0084 PSN 0xa54a70 RKey 0x0000e01 VAddr 0x007fc431156000
GID: 00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:255:255:10:64:04:116
remote address: LID 0000 QPN 0x0084 PSN 0xb35221 RKey 0x0000c01 VAddr 0x007f6376c67000
GID: 00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:255:255:10:64:04:115
---------------------------------------------------------------------------------------
#bytes     #iterations    BW peak[MiB/sec]    BW average[MiB/sec]    MsgRate[Mpps]
Conflicting CPU frequency values detected: 800.000000 != 3600.000000. CPU Frequency is not max.
65536      5000           2758.31             2758.31                0.044133
---------------------------------------------------------------------------------------
``

This confirms that the RDMA network is reachable.

RDMA connectivity test

Procedure

Use the rping utility to verify RDMA communication between the client and server.

The following is an example for verifying RDMA communication from the server side.

root@ceph61:~# rping -s -vVd -C 3 -a 10.64.4.115 -p 20049
count 3
port 20049
created cm_id 0x55adff3546b0
rdma_bind_addr successful
rdma_listen
cma_event type RDMA_CM_EVENT_CONNECT_REQUEST cma_id 0x7f7148000ce0 (child)
child cma 0x7f7148000ce0
created pd 0x55adff34a180
created channel 0x55adff34a1c0
created cq 0x55adff355010
created qp 0x7f7148ff8e90
rping_setup_buffers called on cb 0x55adff3477c0
allocated & registered buffers...
accepting client connection request
cq_thread started.
recv completion
Received rkey 1201 addr 563b1de18750 len 64 from peer
cma_event type RDMA_CM_EVENT_ESTABLISHED cma_id 0x7f7148000ce0 (child)
ESTABLISHED
server received sink adv
server posted rdma read req
rdma read completion
server received read complete
server ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz
server posted go ahead
send completion
recv completion
Received rkey 1101 addr 563b1de18ab0 len 64 from peer
server received sink adv
rdma write from lkey 0x1 laddr 55adff348ab0 len 64
rdma write completion
server rdma write complete
server posted go ahead
send completion
recv completion
Received rkey 1201 addr 563b1de18750 len 64 from peer
server received sink adv
server posted rdma read req
rdma read completion
server received read complete
server ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz
server posted go ahead
send completion
recv completion

Received rkey 1101 addr 563b1de18ab0 len 64 from peer
server received sink adv
rdma write from lkey 0x1 laddr 55adff348ab0 len 64
rdma write completion
server rdma write complete
server posted go ahead
send completion
recv completion
Received rkey 1201 addr 563b1de18750 len 64 from peer
server received sink adv
server posted rdma read req
rdma read completion
server received read complete
server ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz
server posted go ahead
send completion
recv completion
Received rkey 1101 addr 563b1de18ab0 len 64 from peer
server received sink adv
rdma write from lkey 0x1 laddr 55adff348ab0 len 64
rdma write completion
server rdma write complete
server posted go ahead
send completion
cma_event type RDMA_CM_EVENT_DISCONNECTED cma_id 0x7f7148000ce0 (child)
server DISCONNECT EVENT...
wait for RDMA_READ_ADV state 10
rping_free_buffers called on cb 0x55adff3477c0
destroy cm_id 0x55adff3546b0

The following is an example for verifying RDMA communication from the client side.

root@ceph62:~# rping -c -vVd -C 3 -a 10.64.4.115 -p 20049
count 3
port 20049
created cm_id 0x563b1de246b0
cma_event type RDMA_CM_EVENT_ADDR_RESOLVED cma_id 0x563b1de246b0 (parent)
cma_event type RDMA_CM_EVENT_ROUTE_RESOLVED cma_id 0x563b1de246b0 (parent)
rdma_resolve_addr - rdma_resolve_route successful
created pd 0x563b1de1a240
created channel 0x563b1de1a200
created cq 0x563b1de24a50
created qp 0x7f61251d3d90
rping_setup_buffers called on cb 0x563b1de177c0
allocated & registered buffers...
cq_thread started.
cma_event type RDMA_CM_EVENT_ESTABLISHED cma_id 0x563b1de246b0 (parent)
ESTABLISHED
rdma_connect successful
RDMA addr 563b1de18750 rkey 1201 len 64
send completion
recv completion
RDMA addr 563b1de18ab0 rkey 1101 len 64
send completion
recv completion
ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz
RDMA addr 563b1de18750 rkey 1201 len 64
send completion
recv completion
RDMA addr 563b1de18ab0 rkey 1101 len 64
send completion
recv completion
ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz
RDMA addr 563b1de18750 rkey 1201 len 64
send completion
recv completion
RDMA addr 563b1de18ab0 rkey 1101 len 64
send completion
recv completion
ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz
rping_free_buffers called on cb 0x563b1de177c0
cma_event type RDMA_CM_EVENT_DISCONNECTED cma_id 0x563b1de246b0 (parent)
client DISCONNECT EVENT...

This confirms RDMA connectivity before mounting NFS.

What to do next

Mount the NFS shares. For details, see Mounting an NFS share over RDMA.