IBM Support

Accelerate Networking with Shared Memory Communications for IBM Z

News


Abstract

What if we had a networking technology that could provide low latency, high throughput, and save CPU cycles at the same time? This document provides an overview of SMC on a z/OS environment. This document includes:
SMC basics
SMC for Z
SMC in Action
Miscellaneous

Content

SMC Basics
Motivation
The RDMA Approach
Two hosts connecting and showing send and receiving of data
  • RDMA (Remote Direct Memory Access) based technology originating from InfiniBand 
  • Enables a host to read or write directly from or to a remote host’s memory with drastically reduced use of remote host’s CPU (interrupts required for notification only)
  • Native / direct application exploitation requires rewrite of network-related program logic, deep level of expertise in RDMA and a new programming model
  • Therefore, provide a transparent approach:
    • SMC-R: Use RDMA over Converged Ethernet (RoCE) technology
      • Unlike InfiniBand, RoCE does not require unique network components (host adapters, switches, security controls, and so on)
    • SMC-D: Use DMA when both hosts are within a Z system via virtual PCI device
The SMC Protocol
Overview
The SMC-R connection process with an RDMA link between the RDMA devices
For each new TCP connection:
  • Start out with a regular TCP/IP connection, advertising (R)DMA capabilities
  • If peer confirms, negotiate details about the (R)DMA capabilities & connectivity
  • Switch over to an (R)DMA device for actual traffic depending on the peers’ capabilities
  • Regular TCP connection through NICs remains active but idle 
PNET IDs
Two hosts with OSA, RoCE, and HiperSockets and ISM devices with PNET IDs associating devices together
  • PNET ID: Physical network identifier
  • Customer-defined value to logically group NICs and RDMA adapters connected to the same physical network within a host.
  • Defined in
    • IOCDS for any of OSA, RoCE, HiperSockets, or ISM
  • Typically associate
    • OSA and RoCE cards, or
    • HiperSockets and ISM devices
  • Note: PNET IDs help to locate a suitable (R)DMA device for a NIC within a host. The peer can use different PNET IDs (as long as the right devices are grouped)
     
Benefits
  • Less latency
  • Lower CPU usage
  • Run your applications unmodified
    • SMC is transparent to existing applications - no changes required
Heterogeneous Environments
Mixing SMC Usage
Different hosts with different SMC types connecting to a server
  • Both variants of SMC can be used concurrently to provide an optimized solution
  • Enable SMC independent of peers’ capabilities; that is, no commonality in SMC support on all peers required
  • Use:
    • SMC-D for local connections
    • SMC-R for remote connections
    • Fall back to regular TCP where neither SMC variant is supported
       
SMC for Z
SMC-D Overview
An SMC-D connection between two LPARs on a mainframe
  • Intra-CEC connectivity by using Internal Shared Memory (ISM) devices
  • IBM Z hardware requirements
    • IBM z13 (requires driver level 27 (GA2)) and z13s, or later
    • LinuxONE Emperor and LinuxONE Rockhopper, or later
    • Classic mode only (that is, DPM not supported)
  • ISM devices
    • Virtual PCI network adapter of new VCHID type ISM
    • Provides access to memory shared between LPARs
    • 32 ISM VCHIDs per CPC, 255 VFs per VCHID (8K VFs per CPC total) that is, the maximum no. of virtual servers that can communicate over the same ISM VHID is 255
    • Each ISM VCHID represents a unique (isolated) internal network, each having a unique Physical Network ID
      • SMC-D version 2 allows for IP devices without a PNetID
  • PNET ID configuration
    • IOCDS only
    • Use HiperSockets, OSA, or RoCE cards for regular connectivity
SMC-R Overview
An SMC-R connection between two mainframe boxes
  • Cross-CEC connectivity by using RoCE Express cards
  • IBM Z hardware requirements
    • IBM z12EC and z12BC or later
    • LinuxONE Emperor and LinuxONE Rockhopper or later
    • Classic and DPM mode supported
  • RoCE Express cards
    • RoCE Express & RoCE Express2 cards supported
    • Switches need to support and enable Global Pause (standard Ethernet switch flow control feature as described in IEEE 802.3x)
  • PNET ID configuration
    • IOCDS only
    • Use OSA or RoCE card for regular connectivity
  • Note:
    • Linux on Z can use a single RoCE card for regular and RDMA traffic!
    • No link failover!
SMC-R Link Groups
showing an SMC-R link group between two hosts with RoCEs
  • SMC-R link groups provide for load balancing and recovery
    • New TCP connection is assigned to the SMC-R link with the fewest TCP connections
    • Load balancing only performed when multiple RoCE Express adapters are available at each peer
  • Full redundancy requires:
    • Two or more RoCE Express adapters at each peer
    • Unique system internal paths for the RoCE Express adapters
    • Unique physical RoCE switches
  • Partial redundancy still possible in the absence of one or more of these conditions
  • Linux on Z: 
    • No failover support (yet)
SMC in Action
Supported Environments
Operating Systems:
  • SMC-D
    • Linux on Z
    • z/OS: IBM z/OS V2R2 (via APAR) or later
  • SMC-R
    • Linux on Z
    • z/OS: IBM z/OS V2R1 (via APAR) or later
    • AIS: System P with AIX 7.2
Linux on Z Environments
LPAR Yes
z/VM guests Yes (z/VM 6.3 or later)
KVM guests In progress
Docker Limited support, to be improved
Prerequisites
  • Direct connectivity over same IP subnet. That is, no routed traffic, no peers in different IP subnets
  • (R)DMA device(s) attached and configured
  • PNET IDs assigned
  • TCP only, that is, no UDP
  • No IPsec (SSL/TLS works)
  • No NAT (violates same subnet prerequisite)
  • (SMC-D only): Classic mode required (i.e. no DPM support)
Miscellaneous
Summary
Key Attributes
  • Leverages existing Ethernet infrastructure (SMC-R)
  • Transparent to (TCP socket based) application software
  • Preserves existing network addressing-based security models
  • Preserves existing IP topology and network administrative and operational model
  • Transparent to network components such as channel bonding and load balancers
  • Built-in failover capabilities (SMC-R)
Typical Workloads to Benefit
  • Transaction-oriented,
  • Latency-sensitive, and
  • Bulk data streaming, e.g. when running backups.

[{"Type":"MASTER","Line of Business":{"code":"LOB56","label":"Z HW"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SWG90","label":"z\/OS"},"ARM Category":[{"code":"a8m0z0000000AfbAAE","label":"z\/OS Comm Server-\u003ETCPIP-\u003EIF\/DLC - Data Link Control-\u003ESMC-D - Shared Memory Communications Direct"},{"code":"a8m0z0000000AfAAAU","label":"z\/OS Comm Server-\u003ETCPIP-\u003EIF\/DLC - Data Link Control-\u003ESMC-R - Shared Memory Comm. under RDMA"}],"Platform":[{"code":"PF035","label":"z\/OS"}],"Version":"2.3.0;2.4.0;2.5.0;and future releases"}]

Document Information

Modified date:
02 May 2023

UID

ibm16986827

Page Feedback