IT27572: ONMODE -C BLOCK ON RSS NODE CAN REPORT COMMAND HAS TIMED OUT (REGARDLESS OF TIMEOUT VALUE) IF THERE'S LOG ACTIVITY HAPPENING ON

APAR status

• Closed as fixed if next.

Error description

• The onmode -c block command on the RSS node will report the
  following errors:
  
  onmode -c block
  
  The command has timed out.
  onmode could not block server.
  
  In the MSGPATH file you would see something like this:
  
  09:05:54  External Backup preparation complete - Time 15.998
  
  And if no option is given other then -c block the default
  timeout is 15 seconds, so in this case you will see the
  preparation completion time right around the time out value . If
  you ran onmode -c block 120, then the time would be right around
  120...this is because the recovery threads are getting hung up
  and the operation can't complete until the timeout value
  actually kicks in and the command starts to fail, up until that
  happens, the RSS server will mostly just block with the recovery
  threads not allowed to do things and the server then exhausts
  recovery buffer resources and other recovery threads will start
  waiting for empty buffers.
  
  Onstat -g ath output showing xchg threads blocking not being
  able to apply any log records would look like this:
  
  IBM Informix Dynamic Server Version 11.70.FC9 -- Read-Only (RSS)
  -- Up 00:08:44 -- 622188 Kbytes
  Blocked:ARCHIVE_EBR
  
  Threads:
   tid     tcb              rstcb            prty status
  vp-class       name
   42       51abd9e8         4c5473f0         2    cond wait
  packet_con   1cpu         xchg_1.4
   43       51abdd78         4c547c58         2    cond wait
  packet_con   1cpu         xchg_1.5
   44       5477c028         4c5484c0         2    cond wait
  block        1cpu         xchg_1.6
   45       5477c3b8         4c548d28         2    cond wait
  packet_con   1cpu         xchg_1.7
   46       5477c748         4c549590         2    cond wait
  block        1cpu         xchg_1.8
   47       5477cad8         4c549df8         2    cond wait
  block        1cpu         xchg_1.9
   48       55543af0         4c54a660         3    cond wait
  block        1cpu         xchg_2.0
  
  (the cond wait block ones)
  
  Then from onstat -g stk all here's some of the stacks for the
  cond wait block threads and other recovery threads waiting for
  empty buffers before they could contineue to do work:
  
  Stack for thread: 48 xchg_2.0
   base: 0x000000005555f000
    len:   69632
     pc: 0x00000000012ed131
    tos: 0x000000005556fa10
  state: cond wait
     vp: 1
  
  (/usr3/product/1170FC9/bin/oninit) yield_processor_svp
  (/usr3/product/1170FC9/bin/oninit) mt_fast_wait
  (/usr3/product/1170FC9/bin/oninit) critwait
  (/usr3/product/1170FC9/bin/oninit) plogredo
  (/usr3/product/1170FC9/bin/oninit) rlogm_redo
  (/usr3/product/1170FC9/bin/oninit) scan_logredo
  (/usr3/product/1170FC9/bin/oninit) next_lscan
  (/usr3/product/1170FC9/bin/oninit) producer_thread
  (/usr3/product/1170FC9/bin/oninit) startup
  
  (the above is a cond wait block thread)
  
  Stack for thread: 36 bld_logrecs
   base: 0x0000000053f2b000
    len:   69632
     pc: 0x00000000012ed131
    tos: 0x0000000053f3bd10
  state: sleeping
     vp: 1
  
  (/usr3/product/1170FC9/bin/oninit) yield_processor_svp
  (/usr3/product/1170FC9/bin/oninit) mt_yield
  (/usr3/product/1170FC9/bin/oninit) get_empty_Qbufh
  (/usr3/product/1170FC9/bin/oninit) bld_logrecs
  (/usr3/product/1170FC9/bin/oninit) startup
  
  (this is a thread waiting for an empty buffer before it can
  continue to work)
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * Users of IDS 11.70.xC9 and prior versions.                   *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * onmode -c block on RSS node can report command has timed out *
  * (regardless of timeout value) if there's log activity        *
  * happening on the primary.                                    *
  ****************************************************************
  * RECOMMENDATION:                                              *
  ****************************************************************
  

Problem conclusion

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

Applicable component levels