Physical network considerations

The TCP/IP stack must be able to determine which physical network is connected to a particular 10 GbE RoCE Express® Start of change or ISM interface End of change , so that the interface can be associated with the SMC capable IPAQENET, IPAQENET6, IPAQIDIO, and IPAQIDIO6 interfaces that connect to that same physical network. For example, in Figure 1, three distinct and physically separated networks can be accessed by using Start of change SMC-R communications.

Figure 1. Physical networks

The surrounding text describes the figure.

The concept of a physical network identifier (PNetID) was created to simplify this physical network configuration task. With the convention of a PNetID, you can define a value to represent the ID or name of your physical layer 2 LAN fabric or physical broadcast domain. The System z® physical ports that are to be connected to the associated physical networks are then logically associated with their respective PNetIDs. The PNetID then becomes an attribute of the physical port of the feature or adapter, describing how this feature or adapter is physically connected to your data center network. You can specify the PNetID in a single step within the hardware configuration definition (HCD), enabling all operating systems of all associated CPCs to dynamically learn and use this definition.

You can use virtual LANs (VLANs) to logically separate a physical network. If you configure multiple PNetIDs for SMC-R Start of change or SMC-D End of change , then you must ensure that each VLAN or subnet in your configuration does not span more than one PNetID. The physical network that a PNetID represents can include multiple subnets, but each subnet must correspond to a specific PNetID.

Start of change SMC End of change processing requires the use of subnet masks. For more information, see Configuring Shared Memory Communications over RDMA and Configuring Shared Memory Communications - Direct Memory Access.

For more information about the HCD, see z/OS HCD Planning and z/OS HCD User's Guide.

SMC-R physical network considerations

A physically separate IBM® 10 GbE RoCE Express feature is provided to use RDMA over Converged Ethernet (RoCE) on System z. This feature is used with the existing Ethernet connectivity that OSA provides. The 10 GbE RoCE Express feature provides access to the same physical Ethernet fabric that is used for traditional IP connectivity. For more information about the Ethernet switch requirements for RoCE, see Setting up the environment for Shared Memory Communications over RDMA and IBM z Systems™ Planning for Fiber Optic Links.

The operating systems must logically group the associated physical ports of both the 10 GbE RoCE Express and OSA adapters based on their required physical connectivity. Each central processor complex (CPC) connects to a physical network by using both OSA and 10 GbE RoCE Express ports. You can use two ports at most to connect to a physical network at a given time, but you can use as many OSA adapters as necessary for your network bandwidth or usage requirements. An example of this logical grouping, using two OSA adapters and two features, is shown in Figure 1.

One TCP/IP stack can define up to 16 Peripheral Component Interconnect Express (PCIe) function ID (PFID) values. Each PFID value must match a FID value configured in the hardware configuration definition (HCD).

In a dedicated RoCE environment, each PFID represents a unique PCHID definition of a RoCE Express feature, and only one of the two RoCE Express ports for the feature can be used at a time.
In a shared RoCE environment, each PFID represents a virtual function (VF) usage of a RoCE Express feature , and multiple PFID values can be associated with the same physical feature and port.

To match the 10 GbE RoCE Express features with the correct OSA SMC-R capable adapters, you must define a PNetID value for both the 10 GbE RoCE Express interface (physical port) and the corresponding OSA adapters (physical port) within the HCD. The OSA ports correspond to the stack IPAQENET and IPAQENET6 interfaces. VTAM® and the TCP/IP stack then dynamically learn the PNet IDs for the 10 GbE RoCE Express interface and the OSA interfaces when the 10 GbE RoCE Express interface or the OSD interface is started. The 10 GbE RoCE Express interface is associated with only SMC-R capable OSA interfaces that have the same PNetID value defined.

Guideline: The TCP/IP stack does not validate the layer 2 physical network topology or broadcast domain. PNet IDs are values that you assign, and the operating systems learn and use these assigned values but cannot validate them within the Ethernet switched fabric. Therefore, the operating system does not assure or enforce any physical network separation or isolation across different physical networks. To physically isolate unique physical networks, you must ensure that traffic on network A cannot reach hosts on network B.

SMC-D Physical Network considerations

The ISM device does not connect to the same Ethernet fabric that the SMC-D capable OSA interfaces use. However, the operating systems must still logically group the ISM device with the OSA or HiperSockets™ interfaces based on their required physical connectivity. When z/OS® Communications Server activates the first SMC-D capable interface for a given physical network, it also attempts to activate an ISM interface for that same physical network. The Peripheral Component Interconnect Express (PCIe) function ID (PFID) value that represents the ISM device is obtained during activation processing. The possible PFID values to use for ISM devices are configured in the hardware configuration definition (HCD).

To match the ISM device with the correct Start of change SMC-D capable OSA or HiperSockets End of change adapters, you must define a PNetID value for both the ISM device and the corresponding OSA or HiperSockets adapter within the HCD. The OSA ports correspond to the stack IPAQENET and IPAQENET6 interfaces, and the HiperSockets adapters correspond to the stack IPAQIDIO and IPAQIDIO6 interfaces. VTAM and the TCP/IP stack dynamically obtain the PNetIDs for the ISM, OSA, and HiperSockets interfaces when the interface is started. The ISM interface is associated with only SMC-D capable OSA or HiperSockets interfaces that have the same PNetID value defined.

Restriction: The same physical network cannot be used for OSA and HiperSockets interfaces. If you code the same PNet ID for OSA and HiperSockets interfaces for SMC-D communications, the first interface to become fully active will be enabled for SMC-D communications and the second interface will not be able to participate in SMC-D communications.