Rendezvous processing

Shared Memory Communications (SMC) is enabled by using the GLOBALCONFIG statement in the TCP/IP profile data set.

• Shared Memory Communications over RDMA (SMC-R) is enabled by specifying one or more Peripheral Component Interconnect Express® (PCIe) function ID (PFID) values on the SMCR parameter of the GLOBALCONFIG statement in the TCP/IP profile data set. Each PFID value represents an IBM® 10 GbE RoCE Express feature that is configured by using the traditional hardware configuration definition (HCD) tools. TCP/IP activates 10 GbE RoCE Express interfaces when the first SMC-R capable interface is started. SMC-R capable interfaces include IPAQENET or IPAQENET6 interfaces with the OSD channel path ID type. Any TCP connections that are routed over SMC-R capable interfaces are eligible for SMC-R communications.
• Shared Memory Communications - Direct Memory Access (SMC-D) is enabled by specifying the SMCD parameter of the GLOBALCONFIG statement in the TCP/IP profile data set. When SMC-D capable interfaces are activated, z/OS® Communications Server selects an available ISM device that is associated with the same physical network as the SMC-D capable interface. Only one ISM device is activated for each physical network. The ISM device is represented by a Peripheral Component Interconnect Express (PCIe) function ID (PFID) value that is configured by using the traditional hardware configuration definition (HCD) tools. SMC-D capable interfaces include IPAQENET or IPAQENET6 interfaces with the OSD channel path ID type, and IPAQIDIO and IPAQIDIO6 HiperSockets™ interfaces. Any TCP connections that are routed over SMC-D capable interfaces are eligible for SMC-D communications.

The decision about whether an eligible connection uses SMC communications is reached during traditional TCP connection establishment. Rendezvous processing is the term that is used to describe the sequence of connection management flows (shown in Figure 1) that are required to establish SMC-R communications between two peers.

Figure 1. Rendezvous processing

The exchange of information occurs in different stages:

• Extra information in TCP connection establishment flows
  Applications still use the standard three-way handshake mechanisms to establish TCP connections. When SMC-R or SMC-D communications are enabled, the client and the server can indicate support for either SMC-R or SMC-D protocols:

  • The client adds TCP options settings in the SYN request to indicate which protocols that the client supports.
  • The server responds with TCP options settings in the SYN-ACK response to indicate which protocols that the server supports.

  No additional exchange of information is required in this stage of the rendezvous processing.
• In-band SMC Connection Layer Control (CLC) messages

  Conceptually, CLC messages are similar to the SSL handshake processing that occurs after the TCP connection is established.

  SMC-D protocols are preferred over SMC-R protocols. If both the client and the server indicate support for SMC-D processing, after the TCP connection is established, the client and the server negotiate the use of SMC-D for this TCP connection by using SMC-D CLC messages that flow as in-band data over the TCP connection.

  The SMC-D CLC messages exchange the following information:

  • Peer identification information
  • SMC-D buffer information that is necessary to exchange data between the peers

  For detailed description about the information that SMC-D CLC messages exchange, see SMC-D links.

  If the attempt to establish SMC-D communications succeeds, the SMC-D link is established, and the peers can start exchanging data by using the SMC-D buffers.

  If SMC-D communications is not available, and both the client and the server indicate support for SMC-R processing, the client and the server negotiate the use of SMC-R for this TCP connection by using SMC-R CLC messages that flow as in-band data over the TCP connection.

  The SMC-R CLC messages exchange the following information:

  • Network routing information
  • RDMA over Converged Ethernet (RoCE) routing credentials
  • SMC-R buffer information that is necessary to select or establish the RoCE path between the peers

  For detailed description about the information that SMC-R CLC messages exchange, see SMC-R links.

• SMC-R Link Layer Control (LLC) messages

  After the SMC-R information is exchanged, SMC-R LLC messages are exchanged across the RoCE fabric to confirm that the RoCE information is correct and that the remote memory can be accessed. This stage is skipped if an existing RoCE connection is used for this TCP connection.

The TCP/IP stack does not allow the client and server applications to exchange application data during rendezvous processing. Because no application data is exchanged, the TCP connection can revert to IP protocols if there is a failure during the setup of the SMC communications. However, after the TCP connection is committed to using SMC protocols, the TCP connection cannot fall back to using IP protocols if SMC communications encounter an error.

• For SMC-R communications, the TCP connection is committed to use SMC logic after the RoCE connection is confirmed by using the SMC-R LLC messages.
• For SMC-D communications, the TCP connection is committed to use SMC logic after the successful exchange of SMC-D CLC messages.

Both the client and the server nodes maintain a series of rendezvous timers to ensure that the rendezvous processing completes in time. If one of the timed events does not complete as expected, the TCP connection reverts to using IP protocols. However, future TCP connections can still attempt to use SMC.

To reduce the overhead of persistent rendezvous failures (TCP connections reverting to using IP protocols) to the same destination IP address, you can use the default AUTOCACHE function. This function is controlled by the AUTOCACHE and NOAUTOCACHE subparameters of the SMCGLOBAL parameter on the GLOBALCONFIG profile statement and is set to AUTOCACHE by default. The AUTOCACHE function caches rendezvous failures per IP address destination. If the AUTOCACHE function detects too many rendezvous failures to a specific IP address, the function prevents additional rendezvous attempts to that IP address. The AUTOCACHE function is started only when you enable SMC. For more information about enabling SMC, see the description of the GLOBALCONFIG SMCR and SMCD parameters in z/OS Communications Server: IP Configuration Reference.

Even though the data is sent out of band with SMC communications, the TCP connection remains active, primarily to facilitate connection termination processing. If you have TCP server applications that primarily use many short-lived TCP connections, you might want to avoid rendezvous processing. You can prevent these server applications from using SMC in one of the following ways:

Use the AUTOSMC monitoring function

The AUTOSMC monitoring function dynamically monitors incoming TCP connections to local TCP server applications and determines whether the use of SMC is beneficial for the workload. This function is configured by the AUTOSMC and NOAUTOSMC subparameters of the SMCGLOBAL parameter on the GLOBALCONFIG profile statement and is set to AUTOSMC by default. However, the AUTOSMC monitoring function is started only when you enable SMC. For more information about enabling SMC, see the description of the GLOBALCONFIG SMCR and SMCD parameters in z/OS Communications Server: IP Configuration Reference. If the function determines that most of the incoming connections are short-lived and exchange a small amount of data then SMC processing will be bypassed for all new connections to this server. This monitoring is dynamic in nature so that it can detect changes in workload patterns. You can monitor the results of this dynamic monitoring and SMC enablement/disablement using the Netstat ALL/-A command. For more information about the Netstat ALL/-A command, see z/OS Communications Server: IP System Administrator's Commands.

Disable SMC eligiblity for the port

Explicitly specify the NOSMC parameter on the PORT or PORTRANGE statements that define the port or ports that the server application uses. For more information, see z/OS Communications Server: IP Configuration Reference.