Network connections for IBM DB2 Analytics Accelerator

Content

IBM zEnterprise systems offer two ways to set up a high-availibilty connection between a z/OS host and a Netezza host:

• Connecting through customer-managed networking equipment: This method utilizes two 10 GbE OSA cards configured in OSD mode and two customer-procured and managed external switches. The switch ports facing the zEnterprise machines can be long-range (LR) or short-range (SR) ports; the ports facing the Netezza side must be SR ports.
  
• Connecting through the top-of-rack (TOR) switches of an IBM BladeCenter Extension (zBX) managed by the Unified Resource Manager: This type of connection through the zBX allows you set up the connection in one of two ways: either through connecting two 10GbE OSA cards configured in OSD mode to the external ports of the TOR switches, or through using the Intra-Ensemble Data Network (IEDN) of the zBX. In the latter case, the connection is established through existing OSX channels that are managed by a virtual LAN (VLAN). The OSX channels are then used to communicate with both the Netezza hosts and x blade or p blade hosts that reside within the zBX.

To ensure high-availability, set up virtual IP addressing (VIPA) on the z/OS host. VIPA guarantees that IBM DB2 Analytics Accelerator can always connect to DB2 for z/OS, even in case of an OSA card failover. (The Netezza system uses a similar concept: By means of interface bonding, a set of IP addresses is assigned to a virtual IP address called "wall IP".).

The interfaces to the Netezza host must be static interfaces. Bear this in mind, especially if your z/OS host is configured for dynamic routing, such as OSPF, RIP, or RIP 2.

Network setup with OSD channels and customer-managed switches

The following diagram shows a setup with external switches. The setup for a z/OS host and Netezza hosts connecting to zBX TOR switches is almost identical. Just replace the switches in the diagram with a zBX machine.

If you want to use the TOR switches on a zBX rather than external switches, you must create a VLAN and assign the switch ports properly. To continue with the zBX configuration, see Network setup with OSX or OSD channels connected to the zBX Top-of-Rack (TOR) Switches further down in this document.

Configuring OSD channels in the IOCP
Configure the OSD channels in the input/output configuration program (IOCP). See the following example:CHPID PATH=(CSS(0),E1),SHARED,                                *
    PARTITION=((LP1,LP2,LP3,LP4,LP5,LP6,LP7,LP8,LP9),(ZLPA,Z*
    LPB,ZLPC,ZLPD)),PCHID=3A1,TYPE=OSD
CHPID PATH=(CSS(0),E3),SHARED,                                *
    PARTITION=((LP1,LP2,LP3,LP4,LP5,LP6,LP7,LP8,LP9),(ZLPA,Z*
    LPB,ZLPC,ZLPD)),PCHID=5B1,TYPE=OSD                       
CNTLUNIT CUNUMBR=E100,PATH=((CSS(0),E1)),UNIT=OSA              
IODEVICE ADDRESS=(E100,032),CUNUMBR=(E100),UNIT=OSA            
IODEVICE ADDRESS=(E1FE,001),CUNUMBR=(E100),UNIT=OSAD          
CNTLUNIT CUNUMBR=E300,PATH=((CSS(0),E3)),UNIT=OSA
IODEVICE ADDRESS=(E300,032),CUNUMBR=(E300),UNIT=OSA
    IODEVICE ADDRESS=(E3FE,001),CUNUMBR=(E300),UNIT=OSAD

Verifying the OSD channel configuration
Use the display command to display your OSD devices. The console output for each configured channel should be similar to the output for channel path E1 from the previous example:

D M=CHP(E1)
IEE174I 10.06.33 DISPLAY M 797
CHPID E1:  TYPE=11, DESC=OSA DIRECT EXPRESS, ONLINE
DEVICE STATUS FOR CHANNEL PATH E1
     0  1  2  3  4  5  6  7  8  9  A  B  C  D  E  F
0E10 +  +  +  +  +  +  +  +  +  +  +  +  +  +  +  +
0E11 +  +  +  +  +  +  +  +  +  +  +  +  +  +  +  +
0E1F .  .  .  .  .  .  .  .  .  .  .  .  .  .  +  .
SWITCH DEVICE NUMBER = NONE
PHYSICAL CHANNEL ID = 03A1
************************ SYMBOL EXPLANATIONS ************************
+ ONLINE    § PATH NOT VALIDATED   - OFFLINE    . DOES NOT EXIST
              * PHYSICALLY ONLINE   $ PATH NOT OPERATIONAL

Defining ports (VTAM transport list elements or TRLEs) for the TCP/IP configuration
Define a port for each OSD channel as shown in the following example:

TRLE0   VBUILD TYPE=TRL
IPANZA0 TRLE   LNCTL=MPC,                                             *
               READ=E100,                                             *
               WRITE=E101,                                            *
               DATAPATH=E102,                                         *
               PORTNAME=GBE100,                                       *
               MPCLEVEL=QDIO                                            

IPANZA1 TRLE   LNCTL=MPC,                                             *
               READ=E300,                                             *
               WRITE=E301,                                            *
               DATAPATH=E302,                                         *
               PORTNAME=GBE300,                                       *
               MPCLEVEL=QDIO

Verifying the correct assignment of channel paths to ports
To verify a correct assignment of channel paths to ports, run the D NET command for each configured port. See the following example:

D NET,TRL,TRLE=IPANZA0
IST097I DISPLAY ACCEPTED
IST075I NAME = IPANZA0, TYPE = TRLE 605
...
IST2263I PORTNAME = GBE100     PORTNUM =   0   OSA CODE LEVEL = 004E
IST2337I CHPID TYPE = OSD      CHPID = E1
...

Configuring TCP/IP
The following data set (usually called PROFILE TCPIP) shows a working TCP/IP configuration for the previous OSD example:

; STATIC VIPA DEFINITIONS
  DEVICE VIPA  VIRTUAL 0
  LINK VIPAL VIRTUAL 0 VIPA
; Interfaces to Netezza
;Ten Gigabit Interface Definition for Portname GBE100
   INTERFACE TENGBEE1 DEFINE IPAQENET
    PORTNAME GBE100
    IPADDR 10.101.8.110/24
    MTU 8992
    INBPERF DYNAMIC PRIR
    VMAC ROUTEALL
    SOURCEVIPAINT VIPAL
;Ten Gigabit Interface Definition for Portname GBE300
   INTERFACE TENGBEE3 DEFINE IPAQENET
    PORTNAME GBE300
    IPADDR 10.101.8.111/24
    MTU 8992
    INBPERF DYNAMIC PRIR
    VMAC ROUTEALL
    SOURCEVIPAINT VIPAL
HOME
    10.101.8.109  VIPAL

BEGINRoutes
; Destination Subnet Mask First Hop Link Name Packet Size
  ROUTE 10.101.8.0     255.255.255.0   = TENGBEE1 mtu 8992
  ROUTE 10.101.8.0     255.255.255.0   = TENGBEE3 mtu 8992
  ROUTE DEFAULT        9.152.123.1     EC00 mtu defaultsize
ENDRoutes
START TENGBEE1
START TENGBEE3
                                                         
Verifying the TCP/IP configuration
To verify your settings in the TCP/IP configuration data set, run the D TCPIP command as shown in the following example. This way, you can check whether the port names, channel path IDs, IP addresses, TCP/IP route names, and other settings are correct:

D TCPIP,,N,DE
    INTFNAME: TENGBEE1          INTFTYPE: IPAQENET   INTFSTATUS: READY
        PORTNAME: GBE100    DATAPATH: E102     DATAPATHSTATUS: READY
        CHPIDTYPE: OSD
        SPEED: 0000010000
        IPBROADCASTCAPABILITY: NO
        VMACADDR: 0200083B1EAB  VMACORIGIN: OSA  VMACROUTER: ALL
        SRCVIPAINTF: VIPAL
        CFGROUTER: PRI                 ACTROUTER: NON
        ARPOFFLOAD: YES                ARPOFFLOADINFO: YES
        CFGMTU: 8992                   ACTMTU: 8992
        IPADDR: 10.101.8.110/24
        VLANID: NONE                   VLANPRIORITY: DISABLED
        READSTORAGE: GLOBAL (4096K)    INBPERF: DYNAMIC
        CHECKSUMOFFLOAD: YES

        ...

    IPV4 LAN GROUP SUMMARY
      LANGROUP: 00001
        NAME              STATUS      ARPOWNER          VIPAOWNER
        ----              ------      --------          ---------
        TENGBEE3          ACTIVE      TENGBEE3          YES
        TENGBEE1          ACTIVE      TENGBEE1          NO
      OSA-EXPRESS NETWORK TRAFFIC ANALYZER INFORMATION:
      NO OSA-EXPRESS NETWORK TRAFFIC ANALYZER INTERFACES ARE DEFINED

MTU Size Considerations
In order to use the IBM DB2 Analytics Accelerator to accelerate queries, you must use an MTU size of 8992 in the private network end-to-end from System z to the Accelerator, including each System z LPAR connected to the Accelerator, all switches between System z and the Accelerator, and the Accelerator itself. A lower MTU size on all components results in subpar performance and throughput, a different MTU size on one or more components may result in either package fragmentation which will lead to a lower performance, or to situations where packets will not be fragmented, but simply cut off depending on the switch configuration. In the latter case, communication with the Accelerator is not possible in situations where data greater than the maximum segment size (MTU size - headers) is being transmitted. Consequences may be sporadic communication outages, error messages, or stored procedure hangs. To avoid these negative side-effects, it is required to consequently enable jumbo frames with an MTU size of 8992 on all components.
Some scenarios necessitate the use of a smaller MTU size, despite the negative impacts on performance. In such a case, make sure that the MTU size is set to the same size on all components end-to-end: each System z LPAR connected to the Accelerator, all switches between System z and the Accelerator, and the Accelerator itself.
When using customer-managed switches to connect System z to the IBM DB2 Analytics Accelerator, packet fragmentation might result even if the MTU size is set to 8992 on all switches, z/OS LPARs, and the Accelerator. To prevent this, IBM recommends setting the MTU size on the switches - either only for the Accelerator VLAN or globally - to at least 100 bytes over the desired MTU size of 8992 bytes to accommodate for different vendor definitions and implementations of the MTU setting. Furthermore, you should check the switch for oversized and discarded frames to verify that the changes took effect.

Network setup with OSX or OSD channels connected to the zBX Top-of-Rack (TOR) Switches

The IBM zEnterprise Intra Ensemble Data Network (IEDN) infrastructure is based on interconnected, redundant Layer-2 network switching. Included as part of that network are two 10 GbE IEDN TOR switches.

Redundancy in connectivity to the IEDN from System z hosts is achieved through the use of two OSA-Express3 10 GbE cards. The IEDN supports both IP protocol versions 4 and 6 (IPv4 and IPv6).

The network security must be adapted to this type of infrastructure.

Using virtual LANs (VLANs), you can configure network security zones in the IEDN. The network infrastructure is designed in such a way that unauthorized connections to the IEDN are refused.

You define the network infrastructure in the operating system. Access is granted through the Hardware Management Console (HMC).

The default configuration is one VLAN within which multiple VLAN subnets can be defined.

Note: Layer-2 protocols cannot route between different VLANs. To do that, a Layer-3 router is required.

The following diagram shows the connection of a zEnterprise host to two Netezza hosts via the top-of-rack (TOR) switches of an IBM BladeCenter Extension (zBX).

TOR ports to use
Use the external TOR ports J31 through J39 for OSD channel connections. Use the TOR ports J00 through J07 for OSX channel connections through the IEDN.

Configuring OSX channels in the IOCP
Configure the OSX channels in the input/output configuration program (IOCP). See the following example:

CHPID PATH=(CSS(0),E0),SHARED,                                *
    PARTITION=((LP1,LP2,LP3,LP4,LP5,LP6,LP7,LP8,LP9),(ZLPA,Z*
    LPB,ZLPC,ZLPD)),PCHID=3A0,TYPE=OSX
CHPID PATH=(CSS(0),E2),SHARED,                                *
    PARTITION=((LP1,LP2,LP3,LP4,LP5,LP6,LP7,LP8,LP9),(ZLPA,Z*
    LPB,ZLPC,ZLPD)),PCHID=5B0,TYPE=OSX
CNTLUNIT CUNUMBR=E000,PATH=((CSS(0),E0)),UNIT=OSX
IODEVICE ADDRESS=(E000,032),MODEL=X,CUNUMBR=(E000),UNIT=OSA
IODEVICE ADDRESS=(E0FE,001),CUNUMBR=(E000),UNIT=OSAD
CNTLUNIT CUNUMBR=E200,PATH=((CSS(0),E2)),UNIT=OSX
IODEVICE ADDRESS=(E200,032),MODEL=X,CUNUMBR=(E200),UNIT=OSA
IODEVICE ADDRESS=(E2FE,001),CUNUMBR=(E200),UNIT=OSAVerifying the OSX channel configuration
Use the display command to display your OSX devices. The console output for each configured channel should be similar to the output for the channel paths E0 and E2 from the previous example:

D M=CHP(E0)
IEE174I 12.55.58 DISPLAY M 966
CHPID E0:  TYPE=30, DESC=OSA ZBX DATA, ONLINE
DEVICE STATUS FOR CHANNEL PATH E0
     0  1  2  3  4  5  6  7  8  9  A  B  C  D  E  F
0E00 +  +  +  +  +  +  +  +  +  +  +  +  +  +  +  +
0E01 +  +  +  +  +  +  +  +  +  +  +  +  +  +  +  +
0E0F .  .  .  .  .  .  .  .  .  .  .  .  .  .  +  .
SWITCH DEVICE NUMBER = NONE
PHYSICAL CHANNEL ID = 03A0
************************ SYMBOL EXPLANATIONS ************************
+ ONLINE    § PATH NOT VALIDATED   - OFFLINE    . DOES NOT EXIST
* PHYSICALLY ONLINE   $ PATH NOT OPERATIONAL
D M=CHP(E2)
IEE174I 12.56.02 DISPLAY M 968
CHPID E2:  TYPE=30, DESC=OSA ZBX DATA, ONLINE
DEVICE STATUS FOR CHANNEL PATH E2
     0  1  2  3  4  5  6  7  8  9  A  B  C  D  E  F
0E20 +  +  +  +  +  +  +  +  +  +  +  +  +  +  +  +
0E21 +  +  +  +  +  +  +  +  +  +  +  +  +  +  +  +
0E2F .  .  .  .  .  .  .  .  .  .  .  .  .  .  +  .
SWITCH DEVICE NUMBER = NONE
PHYSICAL CHANNEL ID = 05B0
************************ SYMBOL EXPLANATIONS ************************
+ ONLINE    § PATH NOT VALIDATED   - OFFLINE    . DOES NOT EXIST
* PHYSICALLY ONLINE   $ PATH NOT OPERATIONAL

Making z/OS a member of the zEnterprise ensemble
To be able to use the IEDN, the z/OS environment that interacts with the Netezza hosts must be a member of the zEnterprise ensemble.

You can add the member dynamically with the following modify command:

F NET,VTAMOPTS,ENSEMBLE=YES

Console output:

IST097I MODIFY ACCEPTED
IST223I MODIFY COMMAND COMPLETED

You can also add a parameter to your VTAM start options, as in the following example (ensemble parameter in bold print):

CONFIG=00,                     NAME OF ATCCON.. - MEMBER              *
ENSEMBLE=YES,                  Z ENSEMBLE MANAGEMENT ENABLED          *
HOSTSA=&VTAM.,                 SUBAREA NO. OF HOST-SYST.

If the z/OS environment is not a member of the ensemble, the IEDN interface cannot be activated. In this case, the following message is displayed:

EZZ4336I ERROR DURING ACTIVATION OF INTERFACE E000 - CODE 10103037
DIAGNOSTIC CODE 01

Creating the VLAN

1. Start the Hardware Management Console (HMC).
2. Select Manage Virtual Networks.
3. From the Select Action drop-down list, select New Virtual Network.
4. Provide a name and an ID (positive integer) for the VLAN. In the examples that follow, the relevant VLAN is called IDAA and uses the ID 100.
5. Click OK
6. Click Close.

Configuring the TOR switches

1. Start the Hardware Management Console (HMC).
2. Select Configure Top-of-rack (TOR) Switch. This is the HMC module for the assignment of TOR ports to network devices.
3. Select the switch that you want to configure:
   
4. Click OK.
5. Select and assign the ports to configure. See the following example:
   
   The example uses a VLAN named IDAA to connect z/OS with the Netezza hosts. The ID of this VLAN is 100.
   
   All ports of VLAN 100 are in "access" mode. This means that you do not not have to complete further configuration steps in the TCP/IP stack of the z/OS host.
   
   Ports 31 and 32 connect the TOR switch to the Netezza hosts.
   
   Port 39 connects to OSD channel E1, which was introduced with the example in the previous section Configuring OSD channels in the IOCP.
   
   To complete the OSD configuration, repeat these steps to connect OSD channel E3 and the two Netezza hosts to the other TOR switch, preferably by using the same port numbers on the other switch.
   
   The ports that connect to OSX channels cannot be configured here. They run in "trunk" mode, meaning that the ports are assigned internally. Although the screen shot above shows ports in "trunk" mode, the OSX ports are not shown. However, you can view and check the internal assignment under Verifying the OSX channel assignment, as the OSX channels are treated as ensemble members.
   
6. Click OK.
7. Click Close.

Verifying the OSX channel assignment
OSX channels used for IEDN network connections are treated as ensemble members. They are assigned automatically and cannot be configured manually. To check the internal assignment of your OSX channels, complete the following steps:

1. In the HMC, select Manage Virtual Networks.
2. Select the VLAN that you created to connect your z/OS system with the Netezza hosts. Following the example in this document, this would be the IDAA network with VLAN ID 100.
3. From the Select Action drop-down list, select Add hosts to Virtual Network.
4. Select the proper z/OS host. In the example that follows, this is the host called LP1.
5. Click Next.
6. The opening window already shows the OSX channels in the NIC column.
7. Click OK.
8. The new window shows two tabs, Network Information and Members. The Network Information tab is in front. It shows the VLAN name (example: IDAA), a text description, and the VLAN ID (example: 100). Click the Members tab. The OSX channels for each host (if any) are shown in the NIC column. In the example, you find them at the bottom of the list:
   
   
   
9. When finished, click OK and Closed.
   

Defining ports (VTAM transport list elements or TRLEs) for the TCP/IP configuration
This step is not necessary for an IEDN connection with OSX channels. Required VTAM TRLEs are created automatically when needed.

Configuring TCP/IP
The following data set (usually called PROFILE TCPIP) shows a working TCP/IP configuration for the previous OSX example:

; STATIC VIPA DEFINITIONS
  DEVICE VIPA  VIRTUAL 0
  LINK VIPAL VIRTUAL 0 VIPA
; Interfaces to Netezza
; Interfaces to zBX VLAN 100
INTERFACE E000 DEFINE IPAQENET CHPIDTYPE OSX CHPID E0 IPADDR
10.101.8.110/24 VLANID 100 SOURCEVIPAINT VIPAL MTU 8992
INTERFACE E200 DEFINE IPAQENET CHPIDTYPE OSX CHPID E2 IPADDR
10.101.8.111/24 VLANID 100 SOURCEVIPAINT VIPAL MTU 8992
;
; no TRL definition necessary – will be dynamically created
;
HOME
 10.101.8.109  VIPAL
;
BEGINRoutes
; Destination Subnet Mask First Hop Link Name Packet Size
   ROUTE 10.101.8.0     255.255.255.0   = E000 mtu 8992
   ROUTE 10.101.8.0     255.255.255.0   = E200 mtu 8992
   ROUTE DEFAULT        9.152.123.1     EC00 mtu defaultsize
ENDRoutes
START E000
START E200

Verifying the TCP/IP configuration
To verify your settings in the TCP/IP configuration data set, run the D TCPIP command as shown in the following example. This way, you can check whether the port names, channel path IDs, IP addresses, TCP/IP route names, and other settings are correct:

-D TCPIP,,N,DE
 INTFNAME: E000              INTFTYPE: IPAQENET   INTFSTATUS: READY
     PORTNAME: IUTXP0E0  DATAPATH: E002     DATAPATHSTATUS: READY
     CHPIDTYPE: OSX      CHPID: E0
     SPEED: 0000010000
     IPBROADCASTCAPABILITY: NO
     VMACADDR: 024FA3000013   VMACORIGIN: OSA    VMACROUTER: ALL
     SRCVIPAINTF: VLINK
     ARPOFFLOAD: YES                  ARPOFFLOADINFO: YES
     CFGMTU: 8992                     ACTMTU: 8992
     IPADDR: 10.101.8.110/24
     VLANID: 100                      VLANPRIORITY: DISABLED
     DYNVLANREGCFG: NO                DYNVLANREGCAP: YES
     READSTORAGE: GLOBAL (4096K)
     INBPERF: DYNAMIC
     WORKLOADQUEUEING: NO
     CHECKSUMOFFLOAD: YES
     
     ...

IPV4 LAN GROUP SUMMARY
 LANGROUP: 00002
   NAME              STATUS      ARPOWNER          VIPAOWNER
   ----              ------      --------          ---------
   E200              ACTIVE      E200              YES
   E000              ACTIVE      E000              NO

 OSA-EXPRESS NETWORK TRAFFIC ANALYZER INFORMATION:
 NO OSA-EXPRESS NETWORK TRAFFIC ANALYZER INTERFACES ARE DEFINED
 9 OF 9 RECORDS DISPLAYED
 END OF THE REPORT

MTU Size Considerations
When using the zBX TOR switches, the MTU size is set correctly per default which means that the requirements described in the MTU Size Considerations section in the chapter Network setup with OSD channels and customer-managed switches are observed. The task of setting the MTU size correctly is reduced to setting the MTU size on all connected z/OS LPARs and the Accelerator. The latter is done on-site during initial installation by the IBM Support personnel.