Responding to WTO buffer shortages

When WTO message buffer use reaches 80 percent of the limit specified at IPL, the system issues the following message:

IEA405E WTO BUFFER SHORTAGE - 80% FULL

The system also issues a DISPLAY CONSOLES,BACKLOG (D C,B) command to provide information helpful in determining the cause of the buffer shortage.

If the problem continues and WTO buffer use reaches its limit, the system issues the following action message:

IEA404A SEVERE WTO BUFFER SHORTAGE - 100% FULL

When storage for the WTO buffers is exhausted, the system issues the following message:

IEA652A WTO STORAGE EXHAUSTED - WTOS WILL BE DISCARDED

At this point, any new WTOs will be thrown away.

When the system notifies you that the WTO buffers are 80% full, determine the reason for the buffer shortage and correct the problem. Possible reasons are:

A console is not ready and WTO messages are filling the console message buffers because:
- An intervention required condition exists.
- The console has been powered off.
- Some part of the path to the device is not working; for example, an I/O interface is disabled.
- One or more consoles may have their displays held.
A console is not in roll mode, and messages are filling the console message buffers.
A console is in roll or wrap mode but the update time is too long, and messages are filling the console message buffers.
A buffer limit specified at IPL is too low to handle the message traffic in the system. (Either the value on the MLIM parameter in the CONSOLxx member is too low, or the system default for RLIM is too low.)
A program is issuing messages at too rapid a rate and might be in a loop. When a job uses a high percentage of the WTO buffers, the system issues message CNZ3011I which identifies the jobname and the address space.

To determine the extent of the problem and the responsible console or consoles, examine the output from the DISPLAY CONSOLES,BACKLOG (D C,B) command. When messages are backed up for a console, it might be necessary to delete the queue of messages for the console using a CONTROL Q command. You might need to issue CONTROL Q several times to clear the console completely.

When there are too many messages from one job/address space, consider cancelling the job or jobs specified in message CNZ3011I. If cancelling a job would cause a serious impact, look at the messages the job is issuing. If the job seems to be in a loop, then activate an MPF member to suppress or delete the repeating message. Another option is to temporarily remove the message's routing code from all the consoles.

When a high number of buffers is in use for messages from another system in the sysplex, you can route a D C,B command to the other system to determine if a job on the other system is generating too many messages. You can protect your system from a runaway job on another system in the sysplex by using the V CN,DMSCOPE= command.

Figure 1 shows an example of the DISPLAY CONSOLES,BACKLOG output. The system displays information about all consoles, on this system only, that have any outstanding WTO messages. The output in the figure includes the following line:

MSG: CURR=1356  LIM=1500  RPLY:CURR=1    LIM=10    SYS=1        PFK=NONE

In this line, MSG: CURR=1356 LIM=1500 tells you the current use of WTO buffers and the specified limit. RPLY: CURR=1 LIM=1500 tells you the number of WTOR messages that have been displayed and are awaiting operator reply, and the specified limit. The line confirms that more than 80% of the specified WTO buffer limit is reached; 1356 WTO buffers are full and the specified limit is 1500. The display in Figure 1 indicates, through NBUF, the number of buffers queued to each console. In this example, DAVE, with 1217 message buffers filled, is the source of the problem. The buffer limit of 1500 seems adequate, so DAVE is probably failing and causing undisplayed messages to fill the message buffers.

Figure 1. Example of DISPLAY CONSOLES,BACKLOG Command Output

CNZ4100I 17.08.05 CONSOLE DISPLAY    FRAME  1     F      E   SYS=SY1        
CONSOLES MATCHING COMMAND: D C,BACKLOG                                      
MSG:CURR=1356   LIM=1500 RPLY:CURR=1    LIM=10  SYS=SY1      PFK=NONE       
DAVE      TYPE=MCS      STATUS=ACT-SY1                                      
          DEFINED=(SY1)                                                     
          MATCHED=(N/A)                                                     
   ATTRIBUTES ON SY1                                                        
      AUTH=(MASTER)    CMDSYS=*            NBUF=1217                        
      DEV=03E0         LOGON=OPTIONAL      USERID=N/A                       
      MFORM=(S)        AREA=(Z,A)          PFKTAB=01                        
      USE=FC  DEL=RD   RTME=1/4  RNUM=25   SEG=19    CON=N                  
      LEVEL=(ALL)                                                           
      MONITOR=(NONE)                       INTIDS=N  UNKNIDS=N              
      ROUT=(ALL)                                                            
      MSCOPE=(*ALL)                                                         
ADDRESS SPACE WTO BUFFER USAGE                                       
ASID - 0019    JOBNAME = FLOODNUM NBUF =     520                     
MESSAGES COMING FROM OTHER SYSTEMS - WTO BUFFER USAGE                
SYSTEM = 2     NBUF=     4

If the buffer limit is not adequate, issue the CONTROL M,MLIM= command to increase the WTO buffer limit for the duration of the IPL. Your system programmer might code the MLIM parameter on the INIT statement in the CONSOLxx member to raise the WTO buffer limit for the next IPL.

When the number of buffers in use drops below 60% of the limit specified at IPL time, the system issues the following message:

IEA406I WTO BUFFER SHORTAGE RELIEVED

Note:

All lines of an out-of-line multi-line status display that have not been presented occupy message buffers. Therefore, you should erase these displays when they are no longer needed.
The current buffer count can be larger than the specified limit. Even though the buffer count is greater than or equal to the limit, the system always gives a privileged task a buffer unless the storage available for buffers is exhausted.
The system does not use the MLIM and RLIM parameter values specified in the CONSOLxx parmlib member until either the hardcopy medium (SYSLOG or OPERLOG) becomes active or NIP processing is complete. After NIP processing, multiple consoles become active and buffer space becomes important.