Generally, Shared Memory Communications over RDMA (SMC-R) is enabled by specifying one or more Peripheral Component Interconnect Express® (PCIe) function ID (PFID) values on the SMCR parameter of the GLOBALCONFIG statement in the TCP/IP profile data set. Each PFID value represents an IBM® 10GbE RoCE Express feature that is configured by using the traditional hardware configuration definition (HCD) tools. TCP/IP activates 10GbE RoCE Express interfaces when the first SMC-R capable interface is started. SMC-R capable interfaces include IPAQENET or IPAQENET6 interfaces with the OSD channel path ID type. Any TCP connections that are routed over SMC-R capable interfaces are eligible for SMC-R communications.
The decision about whether an eligible connection uses SMC-R communications is reached during traditional TCP connection establishment. Rendezvous processing is the term that is used to describe the sequence of connection management flows (shown in Figure 1) that are required to establish SMC-R communications between two peers.
The exchange of information occurs in three different stages:
- Extra information in TCP connection establishment flows
Applications still use the standard three-way handshake mechanisms to establish TCP connections. When SMC-R communications are enabled, the client adds TCP options settings in the SYN request to indicate that SMC-R protocols are supported. The server, when SMC-R communications are enabled, likewise responds with TCP options settings for SMC-R in the SYN-ACK response. No additional exchange of information is required in this stage of the rendezvous processing.
- In-band SMC-R Connection Layer Control (CLC) messages
After the TCP connection is established, the client and server negotiate the use of SMC-R for this TCP connection by using SMC-R CLC messages that flow as in-band data over the TCP connection. Conceptually, these flows are similar to the SSL handshake processing that occurs after the TCP connection is established.
The CLC messages exchange the following information:
- Network routing information
- RDMA over Converged Ethernet (RoCE) routing credentials
- SMC-R buffer information that is necessary to select or establish the RoCE path between the peers
The information that is exchanged is explained in more detail in SMC-R links.
- SMC-R Link Layer Control (LLC) messages
After the SMC-R information is exchanged, SMC-R LLC messages are exchanged across the RoCE fabric to confirm that the RoCE information is correct and that the remote memory can be accessed. This stage is skipped if an existing RoCE connection is used for this TCP connection.
The TCP/IP stack does not allow the client and server applications to exchange application data during rendezvous processing. Because no application data is exchanged, the TCP connection can revert to IP protocols if there is a failure during the setup of the SMC-R communications. However, after the RoCE connection is confirmed by using the LLC messages, the TCP connection is committed to using SMC-R protocols and cannot fall back to using IP protocols if SMC-R communications encounter an error.
Both the client and the server nodes maintain a series of rendezvous timers to ensure that the rendezvous processing completes in time. If one of the timed events does not complete as expected, the TCP connection reverts to using IP protocols. However, future TCP connections can still attempt to use SMC-R.
Even though the data is sent out of band with SMC-R communications, the TCP connection remains active, primarily to facilitate connection termination processing. If you have TCP server applications that primarily use many short-lived TCP connections, you might want to avoid rendezvous processing. You can prevent these server applications from using SMC-R by specifying the NOSMCR parameter on the PORT or PORTRANGE statements that define the port or ports that the server application uses. For more information, see z/OS Communications Server: IP Configuration Reference.