A fix is available
APAR status
Closed as program error.
Error description
First we see a WRITE DEVICE xxx IS INOPERATIVE Later we see an Abend0c4 in IOSVSSCQ The CSM 4K ECSA is filled up CSM 4K ECSA is full of IUTIL PARMLISTs containing MSG_UDATA_REQ for an OSA that had suffered an INOP on the TRLE's WRITE device 0A01 many days earlier. OSA firmware will "kill" the read to inop the group and that was recognized. The unexpected abend on the write inoped it, but the group never progressed to come down and clean up. VTAM never notified the stack the interface was down, so these QueryARP IP assists built up on the MPNCB found 38K of them on the NCBWORKq (TI buffers) The TI00 buffer pool is really high too. ANALYSIS: PIDS/5752SC1C3 RIDS/NUCLEUS#L RIDS/IOSVSSCQ AB/S00C4 PRCS/00000038 RIDS/IOSVSSCQ#R Symptom Description ------- ----------- PIDS/5752SC1C3 Program id: 5752SC1C3 RIDS/NUCLEUS#L Load module name: NUCLEUS RIDS/IOSVSSCQ Csect name: IOSVSSCQ AB/S00C4 System abend code: 00C4 PRCS/00000038 Abend reason code: 00000038 REGS/918C2 Register/PSW difference for R09:-18C2 RIDS/IOSVSSCQ#R Recovery routine csect name: IOSVSSCQ OTHER SERVICEABILITY INFORMATION Recovery Routine Label: SSCQFRR Date Assembled: 19302 Module Level: UJ00950 Subfunction: IOS SSCH QUEUE ROUTINE Time of Error Information PSW: 04041001 80000000 00000000 010E0ACE Instruction length: 04 Interrupt code: 0038 Failing instruction text: E3202000 00045820 2040C22B Translation exception address: 807AD039_FFB00800 Breaking event address: 00000000_010E0A62 AR/GR 0-1 00000000/000001EF_0299CE20 00000000/000001EF_2EB7C1D8 AR/GR 2-3 00000000/807AD039_FFB00D10 00000000/00000000_025E1E78 AR/GR 4-5 00000000/00000000_2EB7C1D8 00000000/00000000_2EB7C1D8 AR/GR 6-7 00000000/00000000_062EF1F0 00000000/00000000_025E1E78 AR/GR 8-9 00000000/00000000_062ED4A0 00000000/00000000_010E2390 AR/GR 10-11 00000000/00000000_00001BB8 00000000/00000000_2EB7C018 AR/GR 12-13 00000000/000001EF_000019B8 00000000/00000000_062EF288 AR/GR 14-15 00000000/00000000_00000020 00000000/000001EF_00001900
Local fix
From TCP/IP, issue a Vary STOP command against the QDIO interface to cease the TI00 buffer pool and 4K ECSA storage growth that followed the QDIO write control channel INOP.
Problem summary
**************************************************************** * USERS AFFECTED: * * All users of the IBM Communication Server * * for z/OS Version 2 Releases 2, 3 and 4 SNA: * * QDIO * **************************************************************** * PROBLEM DESCRIPTION: * * TI00 Buffer pool and 4K ECSA CSM buffer pool storage growth * * after QDIO write control channel INOP. * * * * IST1222I WRITE DEVICE wwww IS INOPERATIVE, NAME IS trlename * * * **************************************************************** * RECOMMENDATION: * * Apply PTF * **************************************************************** The problem is summarized as follows: 1) A QDIO interface was activated from a TCP/IP stack. 2) At some point, the UCB control block structure for the write control channel was overlaid. The root cause of this overlay is unknown. 3) The overlay results in various Abends in I/O Supervisor code including: ABEND0C4 IN IOSVSSCQ 4) VTAM issues message IST1222I to document the write control channel inop: IST1222I WRITE DEVICE wwww IS INOPERATIVE, NAME IS trlename 5) VTAM should INOP the MPC group from this failure, but failed to do so. 6) A display of the QDIO TRLE whose name is listed in messages IST1222I shows the write subchannel INOP while the read subchannel is ACTIVE: D NET,ID=trlename IST097I DISPLAY ACCEPTED IST075I NAME = trlename, TYPE = TRLE IST486I STATUS= ACTIV, DESIRED STATE= ACTIV IST087I TYPE = LEASED , CONTROL = MPC , HPDT = YES IST1954I TRL MAJOR NODE = trlmajnd IST1715I MPCLEVEL = QDIO MPCUSAGE = SHARE IST2263I PORTNAME = portname PORTNUM = 0 OSA CODE LEVEL = ABCD IST2337I CHPID TYPE = OSD CHPID = C4 PNETID = **NA** IST1577I HEADER SIZE = 4096 DATA SIZE = 0 STORAGE = ***NA*** IST1221I WRITE DEV = wwww STATUS = INOP STATE = ONLINE IST1577I HEADER SIZE = 4092 DATA SIZE = 0 STORAGE = ***NA*** IST1221I READ DEV = rrrr STATUS = ACTIVE STATE = ONLINE 7) Because the MPC group was not INOPed, VTAM failed to notify the stack of the outage. 8) At thirty second intervals, the TCP/IP stack sends two IP Assist commands to query ARP information from the OSA-Express interface. These requests requires two TI00 buffers and two 4K ECSA CSM pages. 9) These are packaged as IUTIL requests and each request occupies 4K of ECSA CSM along with a VTAM TI00 buffer. 10) The IUTIL tipac buffer is queued to the MPNCB NCBWORKQ for processing. Because the write device is INOPed, the data is never processed. This process expects the MPC Group is being brought down because of this condition. 11) Because VTAM failed to INOP the MPC group for this condition, the Write device stays in INOP state and the READ device stays in ACTIVE state from the VTAM perspective. The TCP/IP stack still shows the OSA interfaces as READY. 12) Over time ECSA growth occurs because of this exposure. 13) CSM tracking (CSMUSE) shows the 4K ECSA CSM storage growth is accummulating for component ID CMPID_IUT_Req ('20'X). Also, the TI00 buffer pool is growing too. 14) A TCP/IP stop command must be issued for the OSA express interface in each of the owning TCP/IP stacks to cease the ECSA storage growth.
Problem conclusion
ISTLLCM8 - Has been updated to correctly INOP the MPC group for a QDIO TRLE when an INOP occurs for the write control channel and the read control channel is still active. ISTLLCIE - Has ben included for maintenance purposes.
Temporary fix
Comments
APAR Information
APAR number
OA59569
Reported component name
VTAM MVS/ESA
Reported component ID
569511701
Reported release
230
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2020-05-12
Closed date
2020-05-21
Last modified date
2020-08-03
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
UJ02978 UJ02979 UJ02983
Modules/Macros
ISTLLCM8 ISTLLCIE
Fix information
Fixed component name
VTAM MVS/ESA
Fixed component ID
569511701
Applicable component levels
R220 PSY UJ02978
UP20/07/30 P F007
R230 PSY UJ02979
UP20/07/30 P F007
R240 PSY UJ02983
UP20/07/30 P F007
Fix is available
Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.
[{"Business Unit":{"code":"BU054","label":"Systems w\/TPS"},"Product":{"code":"SSSN3L","label":"z\/OS Communications Server"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"230","Line of Business":{"code":"LOB35","label":"Mainframe SW"}}]
Document Information
Modified date:
05 August 2020