White Papers
Abstract
Preface:
The product name Db2 Analytics Accelerator for z/OS further on is referred to as "the Accelerator" (using the capital "A" as initial character). When talking about setup and functions, we talk about the "accelerator" (with a lower case "a" as initial character) regardless whether the setup is achieved by a single SSC LPAR (single-node) or by a cluster of SSC LPARs (multi-node).
An Accelerator on Z multi-node installation includes the setup of a cluster network for inter-node communication.
Starting with Accelerator V8.1, the cluster network must use the RoCE communication.
The RoCE communication requires
- the Network Express interface when using IBM z17 or IBM LinuxONE Emperor 5 hardware
- the PCI adapter with a RoCE card when using IBM z16, z15, z14 or IBM LinuxONE Emperor 4, III, or II hardware.
With maintenance levels of Accelerator V7.5 prior to 7.5.10, HiperSockets have been used for inter-node communication in a cluster network. When migrating to Accelerator V8.1, it is required to change a HiperSockets-based cluster network to RoCE communication.
The RoCE communication provides the following benefits compared with HiperSockets:
- It consumes significantly less CPU resources than HiperSockets; thus, more of the power of the configured IFLs is used for workload execution.
- It increases the available CPU capacity for workload execution by ~10% compared with HiperSockets.
- It enhances the operational system stability in high workload situations.
This document describes how to
- configure the cluster network of a multi-node installation for RoCE communication using the Network Express interface.
- migrate a cluster network configuration using HiperSockets to one using the RocE communication.
For the configuration of a cluster network with RoCE cards instead of Network Express cards refer to technote https://www.ibm.com/support/pages/node/7031391
Content
For the use of RoCE communication with the Network Express interface in the cluster network, two bonded 25 GbE Network Express cards are recommended. The primary purpose of bonding is to deliver a performance increase to 50 Gb. A secondary purpose is to provide high availability.
Each Network Express card has two ports. Both ports must be connected by using a termination cable to ensure high availability.
As a termination cable, you can use an OM3 or OM4 50-micron multimode fiber optic cable that is terminated with an LC duplex connector.
Multiple accelerators in a single Central Electronic Complex (CEC) can share Network Express cards. It is sufficient to equip the system with two 25GbE Network Express cards and share these cards across multiple multi-node accelerators in the same CEC.
The following figure illustrates an Accelerator on Z multi-node installation using two Network Express cards for the cluster network. For each Network Express card, the ports are terminated with a termination cable. Each accelerator LPAR has access to both Network Express cards.
Within the CEC, the Network Express cards are represented by Function IDs (FIDs), which are assigned to each port of the Network Express card during the hardware configuration. They can be obtained from the I/O Definition File (IODF), the Hardware Management Console (HMC), or the Dynamic Partition Manager (DPM) for each LPAR.
Note: the FIDs are different for each Accelerator LPAR.
Step 1: Add a Network Express card cluster network definition to the accelerator's JSON configuration file
Add a network interface definition for the Network Express cards for each accelerator LPAR (head node, data1 node, ... ) to the "runtime_environments section" of the JSON configuration file. The network interface definition consists of the following information:
- name: The name of the network interface definition. It must be the same for all LPARs, for example "roce".
- ipv4: The IP address of the network interface of this LPAR. The IP address must be different for each LPAR. The IP addresses of all LPARs of a multi-node cluster must all be in the same subnet.
- mode: The bonding mode. Use "balance-rr".
- FID: The function ID of a Network Express card port for the LPAR. Each Network Express card has two ports, and each port has its own FID. The function IDs per Network Express card and port are different for each LPAR.
The following JSON snippet shows an example of an initial configuration of a multi-node accelerator with a cluster network that uses two Network Express cards.
The "network_interface_bindings" section refers to a "cluster_nw" definition named "roce".
The "network_interfaces" section named "roce" contains definitions of four FIDs. That means that two Network Express cards are used each of which has two ports.
The JSON snippet shows the "roce" network interface for the "head" node only.
In the complete JSON file, separate "roce" network interface sections are required for the head node and for all data nodes.
{
"version": "8.1.0",
"accelerator_name": "IDAAONZ",
"accelerator_description": "Accelerator with Network Express cards cluster network",
"accelerator_type": "multi-node",
"db2_pairing_ipv4": "10.20.92.119/24",
"network_interface_bindings": {
"mgmt_nw": "activation-profile",
"db2_nw": "db2",
"cluster_nw": "roce"
},
"runtime_environments": [
{
"cpc_name": "A100",
"head": {
"lpar_name": "LP01",
"network_interfaces": [
{
"name": "db2",
...
},
{
"name": "roce",
"ipv4": "192.168.101.229/24",
"bond_settings": {
"mode": "balance-rr",
"slaves": [
{
"FID": "0000"
},
{
"FID": "0001"
},
{
"FID": "0002"
},
{
"FID": "0003"
}
]
}
}
],
....
},
....
}
...
}If a multi-node accelerator has already been set up, and you want to migrate the cluster network from HiperSockets to RoCE communication with Network Express cards, proceed as follows:
- Do not remove the HiperSockets interface definition so that the "network_interface_bindings" section initially continues to refer to the HiperSockets cluster network.
- Add the network interface definitions for the Network Express cards to each node in the "runtime_environments" section as a currently unused network.
The following JSON snippet shows an example of this scenario. The JSON snippet shows the "roce" network interface for the "head" node only. In the complete JSON file, separate "roce" network interface sections are required for the head node and for all data nodes.
{
"version": "8.1.0",
"accelerator_name": "IDAAONZ",
"accelerator_description": "Accelerator with Network Express cards cluster network",
"accelerator_type": "multi-node",
"db2_pairing_ipv4": "10.20.92.119/24",
"network_interface_bindings": {
"mgmt_nw": "activation-profile",
"db2_nw": "db2",
"cluster_nw": "cluster"
},
"runtime_environments": [
{
"cpc_name": "A100",
"head": {
"lpar_name": "LP01",
"network_interfaces": [
{
"name": "db2",
...
},
{
"name": "cluster",
"ipv4": "192.168.100.229/24",
"device": "0.0.0008"
},
{
"name": "roce",
"ipv4": "192.168.101.229/24",
"bond_settings": {
"mode": "balance-rr",
"slaves": [
{
"FID": "0000"
},
{
"FID": "0001"
},
{
"FID": "0002"
},
{
"FID": "0003"
}
]
}
}
],
....
},
....
}
...
}Step 2: Upload the JSON configuration file to the accelerator
For an initial multi-node accelerator deployment, use the Admin UI to provide a JSON configuration file.
See also https://www.ibm.com/docs/en/daafz/8.1?topic=z-installing-starting-appliance-multi-node-setup.
When the JSON configuration file is complete, upload it using the Admin UI.
To migrate the cluster network of an existing multi-node accelerator from HiperSockets to Network Express cards, upload and apply the updated JSON configuration file in the Admin UI. See also https://www.ibm.com/docs/en/daafz/8.1?topic=z-updating-existing-configuration.
Step 3: Verify the correct configuration of the Network Express cards
After the JSON file has been applied successfully, open the Network panel in the Admin UI and verify that for the head node and for all data nodes, a green status indicator is shown for the "roce" interfaces.
The following sample screenshots show the green status indicator for the "roce" interfaces on the head node and the data nodes.
Additionally, verify that the accelerator nodes can reach each other using the connections of the Network Express card cluster network:
- In the Admin UI, open the Diagnosis panel and select the Test your network tab.
- Starting from the head node, run ping commands to check the Network Express card cluster network IP address of each data node.
- Continue with data1 node. Run a ping command to check the head node and the remaining data nodes by using the cluster network IP address of each node.
- Continue with the remaining data nodes until you have pinged each node from each node on the cluster network IP address.
If one of the ping commands fails and a node cannot be reached, collect a dump; use the Dumps panel in the Admin UI to create the dump for the node that cannot be reached. Open an IBM Support case and attach the dump to the record.
Step 4: Switch to the Network Express card cluster network (only required for a migration from HiperSockets)
Having verified that the Network Express card cluster network is configured correctly, switch from the HiperSockets cluster network to the Network Express card cluster network.
- Edit the JSON configuration file again and change the "network_interface bindings" from "cluster_nw":"cluster" to "cluster_nw":"roce". Note that "cluster" and "roce" are just sample names specified above in Step 2 of the JSON snippet examples. You can use different names in your JSON configuration file.
- Upload and apply the changed JSON configuration file using the Admin UI.
- Perform a Reset of the accelerator (do not select the wipe check box!) in the Admin UI to apply the new Network Express card cluster network across all accelerator nodes. This step incurs a short accelerator outage until all services have been started again. If the reset fails, perform a Shutdown on the Admin UI, followed by a re-activation of all accelerator LPARs to initiate a new restart.
{
"name": "cluster",
"ipv4": "192.168.100.229/24",
"device": "0.0.0008"
},Summary
After all steps have been executed successfully, your multi-node accelerator uses a cluster network with Network Express cards for inter-node communication. That means you can now benefit from the advantages of RoCE communication, such as more IFL resources for workload execution.
[{"Type":"MASTER","Line of Business":{"code":"LOB70","label":"Z TPS"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SS4LQ8","label":"Db2 Analytics Accelerator for z\/OS"},"ARM Category":[{"code":"a8m0z0000000775AAA","label":"Db2 related products and functions-\u003EDb2 Analytics Accelerator for z\/OS"},{"code":"a8m0z000000072oAAA","label":"Install and Migrate"}],"ARM Case Number":"","Platform":[{"code":"PF035","label":"z\/OS"}],"Version":"7.5.0;8.1.0"}]
Was this topic helpful?
Document Information
Modified date:
16 April 2026
UID
ibm17269546