Shared Memory Communications over RDMA (SMC-R)

Edit online

Beginning with IBM® AIX® 7.2 with Technology Level 2, the AIX operating system supports Shared Memory Communications over Remote Direct Memory Access (SMC-R).

SMC-R is a protocol solution that is based on sockets over RDMA and the Internet Engineering Task Force (IETF) Request for Comments (RFC) 7609 publication. It is confined to socket applications by using Transmission Control Protocol (TCP) sockets over IPv4 or IPv6. SMC-R solution enables TCP socket applications to transparently use RDMA, which enables direct, high-speed, low-latency, and memory-to-memory (peer-to-peer) communications.

Communicating peers such as TCP/IP stacks dynamically learn about the shared memory capability by using the traditional TCP/IP connection establishment flows. This process enables the TCP/IP stacks to switch from TCP/IP network flows to an optimized direct memory access flows that use RDMA.

RDMA is available on standard Ethernet-based networks by using the RDMA over Converged Ethernet (RoCE) interface. The RoCE network protocol is an industry-standard initiative by the InfiniBand Trade Association. RoCE interface enables the use of both standard TCP/IP and RDMA solutions such as SMC-R over the same physical local area network (LAN) fabric.

SMC-R protocol solution improves throughput, lowers latency and cost, and maintains existing functions. This model preserves various critical existing operational and network management features of TCP/IP protocol.

SMC-R protocol solution provides failover and load balancing capabilities when multiple RoCE interfaces are configured across both the communicating end points.

The following figure shows SMC-R communication flow between two hosts. By using the TCP option, TCP synchronization operation determines whether both the hosts support SMC-R protocol solution, and then establishes the RoCE network.

SMC-R communication flow
The SMC-R protocol solution is a hybrid solution that can be characterized as follows:
  • The SMC-R protocol solution uses TCP connection (three-way handshake) to establish the SMC-R connection.
  • Each TCP end-point exchanges the TCP option that indicates whether the end point supports the SMC-R protocol. The SMC-R rendezvous information about the reliable connected queue pairs (RC QP) attributes is exchanged within the TCP data stream, which is similar to SSL handshake.
  • When the RC QP connection is established, application data is exchanged through RDMA write operations.
  • TCP connection remains active along with RC QP connection.
The SMC-R protocol solution uses the OpenFabrics Enterprise Distribution (OFED™) core services over RoCE and uses RC QP to transfer data.
SMC-R protocol version 2 interoperability support

The SMC-R protocol solution version 2 supports multiple IP subnets. The SMC-R protocol solution version 1 that is supported in AIX operating system provides interoperability with SMC-R protocol solution version 2 peers that support both versions of the SMC-R protocol solution. The SMC-R protocol solution version 2 peers can be another host to which a TCP/SMC-R connection can be established.

The following versions of AIX operating system support interoperability with SMC-R protocol solution version 2 peers:
  • IBM AIX 7.2 with Technology Level 3 Service Pack 6, and later.
  • IBM AIX 7.2 with Technology Level 4 Service Pack 3, and later.
  • IBM AIX 7.2 with Technology Level 5, and later.