Steps for diagnosing sysplex problems
Use these steps to diagnose sysplex problems. The output is shown in the long, or IPv6-enabled, format.
Procedure
Perform the following steps:
- If subplexing is being used, run the D XCF,GROUP MVS™ command to determine what groups
are being used in the sysplex. Find all the group names with the format
EZBTvvtt.
D XCF,GROUP IXC331I 10.11.09 DISPLAY XCF 637 GROUPS(SIZE): COFVLFNO(2) EZBT11CS(3) EZBT1121(2) EZBT1122(2) EZBT1123(1) ISTCFS11(2) ISTXCF11(2) MVSVIC02(1) MVSVIC11(1) MVSVIC94(1) MVSVIC96(1) MVS031(1) MVS165(1) SYSDAE(9) SYSENF(2) SYSGRS(2) SYSIEFTS(2) SYSIGW00(9) SYSIGW01(9) SYSIKJBC(2) SYSIOS01(2) SYSJES(2) SYSMCS(7) SYSMCS2(4) SYSTTRC(2) SYSWLM(2) VIC140(1)Run the D XCF,GROUP,groupname MVS command for each of the EZBTvvtt format groups to find all the member stacks participating in each subplex group.D XCF,GROUP,EZBT11CS IXC332I 10.11.18 DISPLAY XCF 640 GROUP EZBT11CS: MVS165TCPCS1 MVS165TCPCS3 VIC011TCPCS2 D XCF,GROUP,EZBT1121 IXC332I 10.11.18 DISPLAY XCF 640 GROUP EZBT1121: MVS165TCPCS11 VIC011TCPCS21 D XCF,GROUP,EZBT1122 IXC332I 10.11.18 DISPLAY XCF 640 GROUP EZBT1122: MVS165TCPCS12 VIC011TCPCS22 D XCF,GROUP,EZBT1123 IXC332I 10.11.18 DISPLAY XCF 640 GROUP EZBT1123: VIC011TCPCS23The member names listed are the MVS name concatenated with the TCP/IP stack name. Verify that all the TCP/IP stacks you expect to be communicating are all within the same subplex group.
If the TCP/IP stacks that you expect to be communicating use the same HiperSockets™ CHPID, verify that they have all specified the same IQDVLANID value. Issue the Netstat CONFIG/-f command for each stack and verify that the IQDVLANID value displayed in the Global Configuration Information section of the output is the same for each stack.NETSTAT CONFIG MVS TCP/IP NETSTAT CS V1R8 TCPIP Name: TCPCS 12:55:20 TCP Configuration Table: DefaultRcvBufSize: 00016384 DefaultSndBufSize: 00016384 DefltMaxRcvBufSize: 00262144 MaxReTransmitTime: 120.000 MinReTransmitTime: 0.500 RoundTripGain: 0.125 VarianceGain: 0.250 VarianceMultiplier: 2.000 MaxSegLifeTime: 30.000 DefaultKeepALive: 00000120 DelayAck: Yes RestrictLowPort: Yes SendGarbage: No TcpTimeStamp: Yes FinWait2Time: 600 TTLS Yes UDP Configuration Table: DefaultRcvBufSize: 00065535 DefaultSndBufSize: 00065535 CheckSum: Yes RestrictLowPort: Yes UdpQueueLimit: No IP Configuration Table: Forwarding: Yes TimeToLive: 00064 RsmTimeOut: 00060 IpSecurity: Yes ArpTimeout: 01200 MaxRsmSize: 65535 Format: Short IgRedirect: Yes SysplxRout: No DoubleNop: No StopClawEr: No SourceVipa: No MultiPath: Conn PathMtuDsc: No DevRtryDur: 0000000090 DynamicXCF: Yes IpAddr/PrefixLen: 193.9.200.3/28 Metric: 01 SecClass: 8 IQDIORoute: Yes QDIOPriority: 1 TcpStackSrcVipa: 201.1.10.10 SMF Parameters: Type 118: TcpInit: 01 TcpTerm: 02 FTPClient: 03 TN3270Client: 00 TcpIpStats: 05 Type 119: TcpInit: No TcpTerm: No FTPClient: Yes TcpIpStats: Yes IfStats: Yes PortStats: Yes Stack: Yes UdpTerm: Yes TN3270Client: Yes Global Configuration Information: TcpIpStats: Yes ECSALimit: 0002047K PoolLimit: 2096128K MlsChkTerm: No XCFGRPID: 11 IQDVLANID: 27 Sysplex Monitor: TimerSecs: 60 Recovery: Yes DelayJoin: No AutoRejoin: Yes MONINTF: NO DYNROUTE: NO Network Monitor Configuration Information: PktTrcSrv: Yes TcpCnnSrv: Yes MinLifTim: 3 SmfSrv: Yes Data Trace Setting: JobName: * TrRecCnt: 00000009 Length: FULL IpAddr: * SubNet: * - Run the Netstat VIPADCFG/-F display command
on the distributing stack to confirm that it is configured to distribute
the DVIPA and how it is to be distributed. If the DVIPA
has been deactivated, the deactivated configuration definitions are
displayed under the heading DEACTIVATED DYNAMIC VIPA INFORMATION.
- Figure 1 shows that the TCP/IP identified by TCPCS was configured to distribute DVIPAs. Workload for the first DVIPA, 201.2.10.11 ports 20 and 21, is being distributed to all stacks in the sysplex including TCPCS itself; the configured distribution method is SERVERWLM.
- Workload for 201.2.10.12, ports 20 and 21, is being distributed only to the TCP/IP with dynamic XCF address 193.9.200.2.
- Workload for 201.2.10.13 port 5000 is being distributed to all stacks using the TIMEDAFFinity function.
- Workload for IPv6 DVIPA 2001:0DB8:1::1, port 6000 is being distributed to all stacks; the configured distribution method is SERVERWLM.
- The DVIPA, 201.2.10.23, port 4000, was configured to be distributed to all stacks in the sysplex. Because the DVIPA has been deactivated on this stack, it is not currently being distributed by this stack.
Figure 1. Netstat VIPADCFG/-F exampleD TCPIP,TCPCS,NET,VIPADCFG EZD0101I NETSTAT CS V1R7 TCPCS 876 DYNAMIC VIPA INFORMATION: VIPA BACKUP: IPADDR/PREFIXLEN: 201.2.10.21 RANK: 000080 MOVEABLE: SRVMGR: IPADDR/PREFIXLEN: 201.2.10.22 RANK: 000080 MOVEABLE: SRVMGR: VIPA DEFINE: IPADDR/PREFIXLEN: 201.2.10.11/28 MOVEABLE: IMMEDIATE SRVMGR: NO IPADDR/PREFIXLEN: 201.2.10.12/28 MOVEABLE: IMMEDIATE SRVMGR: NO IPADDR/PREFIXLEN: 201.2.10.13/28 MOVEABLE: IMMEDIATE SRVMGR: NO INTFNAME: DVIPA1 IPADDR: 2001:0DB8:1::1 MOVEABLE: IMMEDIATE SRVMGR: N/A VIPA DISTRIBUTE: DEST: 201.2.10.11..20 DESTXCF: ALL SYSPT: NO TIMAFF: NO FLG: SERVERWLM DEST: 201.2.10.11..21 DESTXCF: ALL SYSPT: NO TIMAFF: NO FLG: SERVERWLM DEST: 201.2.10.12..20 DESTXCF: 193.9.200.2 SYSPT: NO TIMAFF: NO FLG: BASEWLM DEST: 201.2.10.12..21 DESTXCF: 193.9.200.2 SYSPT: NO TIMAFF: NO FLG: BASEWLM DEST: 201.2.10.13..5000 DESTXCF: ALL SYSPT: NO TIMAFF: 45 FLG: BASEWLM DESTINTF: DVIPA1 DEST: 2001:0DB8:1::1..6000 DESTXCF: ALL SYSPT: NO TIMAFF: NO FLG: SERVERWLM Deactivated Dynamic VIPA Information: VIPA Define: IpAddr/PrefixLen: 201.2.10.23/28 Moveable: Immediate SrvMgr: No VIPA Distribute: Dest: 201.2.10.23..4000 DestXCF: ALL SysPt: No TimAff: No Flg: BaseWLM - Run the display Netstat CONFIG/-f command
on the distributing stack and all target stacks to confirm that the
correct IPCONFIG and IPCONFIG6 options have been specified.
Specify SYSPLEXROUTING on the distributor and all target stacks in order to get WLM-based distribution. Verify that DYNAMICXCF was specified on the distributor and all target stacks.
Figure 2 shows the output of this command for the distributing TCP/IP:Figure 2. Netstat CONFIG/-f exampleD TCPIP,TCPCS,N,CONFIG EZD0101I NETSTAT CS V1R8 TCPCS 928 TCP CONFIGURATION TABLE: DEFAULTRCVBUFSIZE: 00016384 DEFAULTSNDBUFSIZE: 00016384 DEFLTMAXRCVBUFSIZE: 00262144 MAXRETRANSMITTIME: 120.000 MINRETRANSMITTIME: 0.500 ROUNDTRIPGAIN: 0.125 VARIANCEGAIN: 0.250 VARIANCEMULTIPLIER: 2.000 MAXSEGLIFETIME: 30.000 DEFAULTKEEPALIVE: 00000120 DELAYACK: YES RESTRICTLOWPORT: YES SENDGARBAGE: NO TCPTIMESTAMP: YES FINWAIT2TIME: 600 UDP CONFIGURATION TABLE: DEFAULTRCVBUFSIZE: 00065535 DEFAULTSNDBUFSIZE: 00065535 CHECKSUM: YES RESTRICTLOWPORT: YES UDPQUEUELIMIT: YES IP CONFIGURATION TABLE: FORWARDING: YES TIMETOLIVE: 00064 RSMTIMEOUT: 00060 ARPTIMEOUT: 01200 MAXRSMSIZE: 65535 FORMAT: LONG IGREDIRECT: YES SYSPLXROUT: YES DOUBLENOP: NO STOPCLAWER: NO SOURCEVIPA: YES MULTIPATH: NO PATHMTUDSC: NO DEVRTRYDUR: 0000000090 DYNAMICXCF: YES IPADDR/PREFIXLEN: 193.15.1.1/24 METRIC: 02 IQDIOROUTE: NO TCPSTACKSRCVIPA: 203.15.1.1 IPV6 CONFIGURATION TABLE: FORWARDING: YES HOPLIMIT: 00255 IGREDIRECT: YES SOURCEVIPA: YES MULTIPATH: NO ICMPERRLIM: 00003 IGRTRHOPLIMIT: NO DYNAMICXCF: YES IPADDR: 2001:0DB8::151:0 INTFID: 0006:0007:0008:0009 TCPSTACKSRCVIPA: DVIPA1 SMF PARAMETERS: TYPE 118: TCPINIT: 00 TCPTERM: 00 FTPCLIENT: 00 TN3270CLIENT: 00 TCPIPSTATS: 00 TYPE 119: TCPINIT: NO TCPTERM: NO FTPCLIENT: NO TCPIPSTATS: NO IFSTATS: NO PORTSTATS: NO STACK: NO UDPTERM: NO TN3270CLIENT: NO GLOBAL CONFIGURATION INFORMATION: TCPIPSTATS: NO ECSALIMIT: 0000000K POOLLIMIT: 0000000K MLSCHKTERM: NO SYSPLEX MONITOR: TIMERSECS: 0060 RECOVERY: YES DELAYJOIN: NO AUTOREJOIN: YES MONINTF: NO DYNROUTE: NORun the D WLM,SYSTEMS command on the distributing stack and all targets stack to confirm that WLM is active. For more information about the DISPLAY command, see z/OS MVS System Commands. Figure 3 shows an example:Figure 3. D WLM,SYSTEMS exampleD WLM,SYSTEMS IWM025I 16.38.58 WLM DISPLAY 963 ACTIVE WORKLOAD MANAGEMENT SERVICE POLICY NAME: DEFAULT ACTIVATED: 2003/10/29 AT: 14:51:50 BY: N/A FROM: VIC015 DESCRIPTION: IBM'S WLM DEFAULT POLICY RELATED SERVICE DEFINITION NAME: N/A INSTALLED: 2003/10/29 AT: 14:51:50 BY: N/A FROM: N/A WLM VERSION LEVEL: LEVEL013 WLM FUNCTIONALITY LEVEL: LEVEL001 STRUCTURE SYSZWLM_WORKUNIT STATUS: DISCONNECTED *SYSNAME* *MODE* *POLICY* *WORKLOAD MANAGEMENT STATUS* VIC015 GOAL DEFAULT ACTIVE, NOT RUNNING WITH ACTIVE POLICY - Run the display Netstat VIPADYN/-v command
on the distributing stack to verify that the DVIPA status is ACTIVE
and the distribution status is DIST or DIST/DEST. The deactivated
DVIPA 203.2.10.23 do not appear in this display. Figure 4 shows an example: Figure 4. Netstat VIPADYN/-v example
D TCPIP,TCPCS,NET,VIPADYN EZD0101I NETSTAT CS V1R8 TCPCS 827 IPADDR/PREFIXLEN: 201.2.10.11/28 STATUS: ACTIVE ORIGIN: VIPADEFINE DISTSTAT: DIST/DEST ActTime: 03/02/2005 16:45:20 IPADDR/PREFIXLEN: 201.2.10.12/28 STATUS: ACTIVE ORIGIN: VIPADEFINE DISTSTAT: DIST ActTime: 03/02/2005 16:45:20 IPADDR/PREFIXLEN: 201.2.10.13/28 STATUS: ACTIVE ORIGIN: VIPADEFINE DISTSTAT: DIST/DEST ActTime: 03/02/2005 16:45:20 IPADDR/PREFIXLEN: 201.2.10.21 STATUS: BACKUP ORIGIN: VIPABACKUP DISTSTAT: ActTime: n/a IPADDR/PREFIXLEN: 201.2.10.22 STATUS: BACKUP ORIGIN: VIPABACKUP DISTSTAT: ActTime: n/a INTFNAME: DVIPA1 IPADDR: 2001:0DB8:1::1 STATUS: ACTIVE ORIGIN: VIPADEFINE DISTSTAT: DIST/DEST ActTime: 03/02/2005 16:45:20 - Run display Netstat VIPADYN/-v command on the target stacks
to verify that they have activated the distributed DVIPA and have
it designated as a DEST. In this case, TCPCS2 has designated the distributed
DVIPAs as DEST and TCPCS2 is a backup stack for several DVIPAs (status
and origin show backup). Figure 5 shows
an example: Figure 5. Netstat VIPADyn/-v example
D TCPIP,TCPCS2,NET,VIPADYN EZD0101I NETSTAT CS V1R8 TCPCS2 905 IPADDR/PREFIXLEN: 201.2.10.11/28 STATUS: BACKUP ORIGIN: VIPABACKUP DISTSTAT: DEST ActTime: 03/02/2005 16:45:20 IPADDR/PREFIXLEN: 201.2.10.12/28 STATUS: BACKUP ORIGIN: VIPABACKUP DISTSTAT: DEST ActTime: 03/02/2005 16:45:20 IPADDR/PREFIXLEN: 201.2.10.13/28 STATUS: ACTIVE ORIGIN: DISTSTAT: DEST ActTime: 03/02/2005 16:45:20 IPADDR/PREFIXLEN: 201.2.10.21 STATUS: BACKUP ORIGIN: VIPABACKUP DISTSTAT: ActTime: n/a IPADDR/PREFIXLEN: 201.2.10.22 STATUS: BACKUP ORIGIN: VIPABACKUP DISTSTAT: ActTime: n/a INTFNAME: DVIPA1 IPADDR: 2001:0DB8:1::1 STATUS: ACTIVE ORIGIN: DISTSTAT: DEST ActTime: 03/02/2005 16:45:20 - Run the Sysplex VIPADyn command from any
stack in the sysplex to get a global view of how and where DVIPAs
are defined within the sysplex and what their status is on each stack.
Deactivated DVIPA configurations do not appear in this display. Figure 6 shows the following:
- Which TCP/IPs own distributed DVIPAs, DIST field=BOTH or DIST
- Which TCP/IPs have been made targets, DIST field = DEST
- The status of all other DVIPAs in this sysplex
Figure 6. Sysplex VIPADyn exampleD TCPIP,TCPCS2,SYSPLEX,VIPAD EZZ8260I SYSPLEX CS V1R7 948 VIPA DYNAMIC DISPLAY FROM TCPCS2 AT VIC015 IPADDR: 201.2.10.11 ORIGIN: VIPABACKUP TCPNAME MVSNAME STATUS RANK DIST -------- -------- ------ ---- ---- TCPCS VIC015 ACTIVE BOTH TCPCS2 VIC015 BACKUP 100 DEST TCPCS3 VIC015 BACKUP 010 DEST IPADDR: 201.2.10.12 ORIGIN: VIPABACKUP TCPNAME MVSNAME STATUS RANK DIST -------- -------- ------ ---- ---- TCPCS VIC015 ACTIVE DIST TCPCS2 VIC015 BACKUP 075 DEST TCPCS3 VIC015 BACKUP 010 IPADDR: 201.2.10.13 TCPNAME MVSNAME STATUS RANK DIST -------- -------- ------ ---- ---- TCPCS VIC015 ACTIVE BOTH TCPCS3 VIC015 BACKUP 010 DEST TCPCS2 VIC015 ACTIVE DEST IPADDR: 201.2.10.21 ORIGIN: VIPABACKUP TCPNAME MVSNAME STATUS RANK DIST -------- -------- ------ ---- ---- TCPCS3 VIC015 ACTIVE TCPCS2 VIC015 BACKUP 100 TCPCS VIC015 BACKUP 080 IPADDR: 201.2.10.22 ORIGIN: VIPABACKUP TCPNAME MVSNAME STATUS RANK DIST -------- -------- ------ ---- ---- TCPCS3 VIC015 ACTIVE TCPCS VIC015 BACKUP 080 TCPCS2 VIC015 BACKUP 075 INTFNAME: DVIPA1 IPADDR: 2001:0DB8:1::1 TCPNAME MVSNAME STATUS RANK DIST -------- -------- ------ ---- ---- TCPCS VIC015 ACTIVE BOTH TCPCS2 VIC015 ACTIVE DEST - Run
the Netstat VDPT/-O command on the distributing stack to confirm that
there are target stacks available with server applications ready.
With the keyword DETAIL you can also see the following fields. For a complete DETAIL display example, see z/OS Communications Server: IP System Administrator's Commands.
- Raw (before normalization) WLM composite weight
- The original CP, zAAP, and zIIP weights, and the proportioned CP, zAAP, and zIIP weights that were used to determine the raw WLM composite weight. The original CP, zAAP, and zIIP weights, and the proportioned CP, zAAP, and zIIP weights are for only SERVERWLM and BASEWLM distribution algorithms.
- The target server connection responsiveness factors that make up the TSR and the current WLM or QOS weight for each service level mapping to a DVIPA or port entry for each target stack (each DESTXCF ADDR). See the z/OS Communications Server: IP User's Guide and Commands for more information.
This display shows only target stacks that are currently up and have joined the sysplex. If there are fewer entries than what resulted from the display d tcpip,,net,vipadcfg command, the missing entries might be for target stacks that are not yet up, or for stacks that are already up now, but that do not specify the expected dynamic XCF address. Figure 7 shows an example:D TCPIP,TCPCS,NET,VDPT,DETAIL EZD0101I NETSTAT CS V1R7 TCPCS 010 DYNAMIC VIPA DESTINATION PORT TABLE: DEST:201.2.10.11..20 DESTXCF:193.9.200.2 TOTALCONN:0000000959 RDY:001 WLM:11 TSR: 79 FLG:SERVERWLM TCSR: 100 CER: 75 SEF: 79 Weight 44 Raw CP: 44 zAAP: 00 zIIP: 00 Proportional CP: 44 zAAP: 00 zIIP: 00 QosPlcAct:*DEFAULT* W/Q:01 DEST:201.2.10.11..20 DESTXCF:193.15.1.1 TOTALCONN:0000000330 RDY:001 WLM:9 TSR: 68 FLG:SERVERWLM TCSR: 87 CER: 75 SEF: 79 Weight 36 Raw CP: 50 zAAP: 00 zIIP: 34 Proportional CP: 05 zAAP: 00 zIIP: 31 QosPlcAct:*DEFAULT* W/Q:01 DEST:201.2.10.11..20 DESTXCF:193.15.3.1 TOTALCONN:0000000315 RDY:001 WLM:15 TSR: 100 FLG:SERVERWLM TCSR: 100 CER: 100 SEF: 100 Weight 60 Raw CP: 60 zAAP: 00 zIIP: 00 Proportional CP: 60 zAAP: 00 zIIP: 00 QosPlcAct:*DEFAULT* W/Q:01 DEST:201.2.10.11..21 DESTXCF:193.9.200.2 TOTALCONN:0000000021 RDY:001 WLM:15 TSR: 100 FLG:SERVERWLM TCSR: 100 CER: 100 SEF: 100 Weight 60 Raw CP: 60 zAAP: 00 zIIP: 00 Proportional CP: 60 zAAP: 00 zIIP: 00 QosPlcAct:*DEFAULT* W/Q:01 DEST:201.2.10.11..21 DESTXCF:193.15.1.1 TOTALCONN:0000000008 RDY:001 WLM:11 TSR: 78 FLG:SERVERWLM TCSR: 99 CER: 99 SEF: 80 Weight 44 Raw CP: 44 zAAP: 00 zIIP: 00 Proportional CP: 44 zAAP: 00 zIIP: 00 QosPlcAct:*DEFAULT* W/Q:01 DEST:201.2.10.11..21 DESTXCF:193.15.3.1 TOTALCONN:0000000007 RDY:001 WLM:10 TSR: 94FLG:SERVERWLM TCSR: 97 CER: 98 SEF: 97 Weight 40 Raw CP: 40 zAAP: 00 zIIP: 00 Proportional CP: 40 zAAP: 00 zIIP: 00 QosPlcAct:*DEFAULT* W/Q:01 DEST:201.2.10.12..20 DESTXCF:193.9.200.2 TOTALCONN:0000000000 RDY:001 WLM:03 TSR: 99 FLG:BASEWLM TCSR: 100 CER: 99 SEF: 99 Weight 12 Raw CP: 20 zAAP: 11 zIIP: 00 Proportional CP: 02 zAAP: 10 zIIP: 00 QosPlcAct:*DEFAULT* W/Q:01 DEST:201.2.10.12..21 DESTXCF:193.9.200.2 TOTALCONN:0000000000 RDY:001 WLM:03 TSR: 100 FLG:BASEWLM TCSR: 100 CER: 100 SEF: 100 Weight 12 Raw CP: 12 zAAP: 00 zIIP: 00 Proportional CP: 12 zAAP: 00 zIIP: 00 QosPlcAct:*DEFAULT* W/Q:01 DEST:201.2.10.13..5000 DESTXCF:193.9.200.2 TOTALCONN:0000000000 RDY:001 WLM:03 TSR: 0 FLG:BASEWLM TCSR: 90 CER: 75 SEF: 0 Weight 12 Raw CP: 12 zAAP: 00 zIIP: 00 Proportional CP: 12 zAAP: 00 zIIP: 00 QosPlcAct:*DEFAULT* W/Q:01 DEST:201.2.10.13..5000 DESTXCF:193.15.1.1 TOTALCONN:0000000000 RDY:001 WLM:01 TSR: 27 FLG:BASEWLM TCSR: 100 CER: 27 SEF: 27 Weight 04 Raw CP: 04 zAAP: 00 zIIP: 00 Proportional CP: 04 zAAP: 00 zIIP: 00 QosPlcAct:*DEFAULT* W/Q:01 DEST:201.2.10.13..5000 DESTXCF:193.15.3.1 TOTALCONN:0000000000 RDY:001 WLM:01 TSR: 48 FLG:BASEWLM TCSR: 75 CER: 64 SEF: 64 Weight 04 Raw CP: 04 zAAP: 00 zIIP: 00 Proportional CP: 04 zAAP: 00 zIIP: 00 QosPlcAct:*DEFAULT* W/Q:01 DESTINTF:DVIPA1 DEST:1::1..6000 DESTXCF:FEC0::151:0 TOTALCONN:0000000511 RDY:001 WLM:11 TSR: 79FLG:SERVERWLM TCSR: 99 CER: 98 SEF: 80 Weight 44 Raw CP: 44 zAAP: 00 zIIP: 00 Proportional CP: 44 zAAP: 00 zIIP: 00 QosPlcAct:*DEFAULT* W/Q:01 DESTINTF:DVIPA1 DEST:1::1..6000 DESTXCF:FEC0::152:0 TOTALCONN:0000001410 RDY:001 WLM:10 TSR: 100 FLG:SERVERWLM TCSR: 100 CER: 95 SEF: 100 Weight 40 Raw CP: 40 zAAP: 00 zIIP: 00 Proportional CP: 40 zAAP: 00 zIIP: 00 QosPlcAct:*DEFAULT* W/Q:01Figure 7. Netstat VDPT/-O example - Examine the READY (RDY) count fields. The READY (RDY) count
is the number of servers that are currently listening on the DVIPA
and PORT specified in the DEST: field on the target stack that was
identified by the DESTXCF address.
For servers that use more than one port, the RDY value reflects the port where a LISTEN is performed. For example, for FTP, the control connection port (port 21) is where the RDY count is greater than 0. If the ready count is not as expected, proceed to step 11 to verify whether any non-quiesced server is listening on the DPORT on the target stack. If there is a server listening on the target stack, verify that it has not been quiesced by a VARY TCPIP,,SYSPLEX,QUIESCE command. On the target stack, run the Netstat ALL/-A command and verify that the quiesced value is NO.
- Check the TotalConn count to see the distribution history.
This is a cumulative count of the number of connections that have
been forwarded by the distributing stack to each target stack. If the connections are not being distributed to the target stacks as expected and either the WLM field or the W/Q field contains 00, then consider the following:
- If using WLM to distribute connections based upon the workload of the target stacks, verify that all participating stacks (the distributor and all targets) have SYSPLEXROUTING specified. See step 3 for instructions for verifying this. Also, verify that WLM is configured and active on all participating stacks. See step 3.
- If the WLM configuration appears correct and BASEWLM is being
used as the distribution method, consider whether the unexpected distribution
results might be caused by the current workload on the target stacks.
If SERVERWLM is being used, consider whether the unexpected distribution results might be the result of how well the server is meeting the goals of its service class, and the amount of workload available on the system given the importance of its service class. See the z/OS Communications Server: IP Configuration Guide for an overview of how WLM determines server recommendations and how they are used by TCP/IP. For more detailed information about sysplex routing services, see z/OS MVS Planning: Workload Management.
- If some entries have a low TSR value, consider whether network
or server performance problems might be affecting distribution. Examine
the TCSR, CER, and SEF values in the DETAIL output for these entries.
- A low TCSR value indicates a connectivity problem between the sysplex distributor stack and the target stack for those particular DVIPA, Port, and Destination entries. To diagnose, verify connectivity across the dynamic XCF interface (HiperSockets, IUTSAMEH, XCF interfaces) or that VIPAROUTE is working properly by interface-specific ping.
- If the SEF value is low but the CER is not, then the application
on this target is having problems with accepting new connections.
Take the following steps to diagnose:
- Verify that the application is accepting connections by connecting to another IP address on the target.
- If the connection is not accepted, verify that the application is getting dispatched correctly.
- If the connection is accepted, verify that the application can properly handle the load (amount of work, connections, tasks).
- If the SEF and CER values are low, then the target stack is having
problems with establishing connections with one or more clients. Take
the following steps to diagnose:
- Find the client to which the connection is failing. Because netstat conn does not show connections in syn sent, the best way to find the clients that are not completing the three-way handshake is a packet trace (other means of finding the client IP are possible).
- If the client is known, verify that the route from the target to the client is correct by reviewing the target stack routing table. If the route from the target to the client is correct, verify that traceroute shows the correct path is being followed to the client.
- If the route from the target to the client is correct, verify that traceroute shows the correct path is being followed to the client.
- If all entries representing distribution to the same target are very low, or 0, this might indicate that the target stack is experiencing problems.
- If you used a VIPAROUTE definition to specify the route from the distributor to the target, check the specified route to verify that it is active.
- If SERVERWLM is being used as the distribution method and a server has a WLM weight of 0, verify that the server is using the appropriate WLM Policy and that the system is not too overloaded to enable the server to meet its policy goals. See z/OS Communications Server: IP Configuration Guide for an overview of how WLM determines server recommendations and how they are used by TCP/IP. For more detailed information about sysplex routing services, see z/OS MVS Planning: Workload Management.
- If the unexpected distribution results have not yet been explained and Sysplex Distributor Performance Policies have been defined using Policy Agent, consider whether the distribution might be caused by two network performance issues (TCP retransmissions and timeouts).
- If Sysplex Distributor Routing Policies have been defined using Policy Agent, consider whether the definition of that policy is affecting the connection distribution. After determining which connections are not being distributed correctly, run D TCPIP,TCPCS,NET,VCRT,DETAIL (see step 10) to determine the policy action to which each connection maps. Look at the QoS weights for those policy actions in the VDPT DETAIL display to see whether they are unusually low. The Policy Agent log on the target stack can display for each DVIPA/Port the QoS service level fractions used to modify the QoS weight. It can also display the calculations that caused a QoS fraction to be set abnormally high (such as connection limit exceeded or throughput exceeded). See Diagnosing Policy Agent problems for more information.
- Run the Netstat VCRT/-V command on the distributing
stack to check whether there are any active connections that are being
routed by the distributor. If you run the command with the keyword
DETAIL (d tcpip,tcpcs,net,vcrt,detail) you can
see the policy rule and policy action that each connection maps to.
If the VCRT table shown in Figure 8 is empty, then connection requests might not be reaching the distributor. Check for a routing problem from the client to the distributor.
If you see expected entries in the table, note the dynamic XCF address and proceed to step 11. Figure 8 shows an example:Figure 8. d tcpip,tcpcs,net,vcrt,detail exampleD TCPIP,TCPCS,NET,VCRT,DETAIL EZD0101I NETSTAT CS V1R7 TCPCS 758 DYNAMIC VIPA CONNECTION ROUTING TABLE: DEST: 201.2.10.11..21 SOURCE: 203.110.1.1..1031 DESTXCF: 193.15.1.1 POLICYRULE: *NONE* POLICYACTION: *NONE* DEST: 201.2.10.12..21 SOURCE: 203.110.1.1..1033 DESTXCF: 193.9.200.2 DEST: 201.2.10.13..5000 SOURCE: 203.110.1.1..0 DESTXCF: 193.15.1.1 CFGTIMAFF: 0045 TIMAFFCNT: 0000000002 TIMAFFLFT: 0000 DEST: 201.2.10.13..5000 SOURCE: 203.110.1.1..1029 DESTXCF: 193.15.1.1 POLICYRULE: *NONE* POLICYACTION: *NONE* DEST: 201.2.10.13..5000 SOURCE: 203.110.1.1..1030 DESTXCF: 193.15.1.1 POLICYRULE: *NONE* POLICYACTION: *NONE* - Go to the target stacks represented by the
DESTXCF ADDR field in the VCRT or VDPT display and run the Netstat
ALLCONN(/-a),IPA=201.2.10.12 display command to see the connections
on the target stack. Figure 9 shows an
example: Figure 9. Netstat ALLConn/-a example with IPAddr/-I filter value of 201.2.1.12
D TCPIP,TCPCS2,NET,ALLCONN,IPA=201.2.10.12 EZD0101I NETSTAT CS V1R7 TCPCS2 846 USER ID CONN STATE FTPD1 000000F3 ESTBLSH LOCAL SOCKET: ::FFFF:201.2.10.12..21 FOREIGN SOCKET: ::FFFF:203.110.1.1..1033 1 OF 1 RECORDS DISPLAYEDTip: For a variety of reasons, the VCRT and ALLCONN displays might not match exactly. For example, with short-lived connections such as Web connections, an entry might show up in one display but be gone by the time the second display is run. Also, the distributing stack places an entry into the Dynamic VIPA Connection Routing Table when it first forwards a connection request. A busy server might reject these connection requests, and therefore cause a temporary mismatch in the two displays.
Parent topic: Diagnosing dynamic VIPA and sysplex problems