IT41566: THE MONREPLCLITHREAD IS HOLDING AN EXCESSIVE NUMBER OF NODE LOCKS WHICH COULD LEAD TO LOCKING ISSUES

8.1.15.100-IBM-SPSRV-WindowsX64
8.1.15.100-IBM-SPSRV-Linuxx86_64
8.1.15.100-IBM-SPSRV-Linuxs390x
8.1.15.100-IBM-SPSRV-Linuxppc64le
8.1.15.100-IBM-SPSRV-AIX
8.1.16.000-IBM-SPCMS-WindowsX64
8.1.16.000-IBM-SPCMS-WindowsI32
8.1.16.000-IBM-SPCMS-Linuxx86_64
8.1.16.000-IBM-SPOC-WindowsX64
8.1.16.000-IBM-SPOC-Linuxx86_64
8.1.16.000-IBM-SPOC-Linuxs390x
8.1.16.000-IBM-SPOC-LinuxPPC64le
8.1.16.000-IBM-SPOC-AIX
8.1.16.000-IBM-SPSRV-WindowsX64
8.1.16.000-IBM-SPSRV-Linuxx86_64
8.1.16.000-IBM-SPSRV-Linuxs390x
8.1.16.000-IBM-SPSRV-Linuxppc64le
8.1.16.000-IBM-SPSRV-AIX
8.1.14.200-IBM-SPSRV-WindowsX64
8.1.14.200-IBM-SPSRV-Linuxx86_64
8.1.14.200-IBM-SPSRV-Linuxs390x
8.1.14.200-IBM-SPSRV-Linuxppc64le
8.1.14.200-IBM-SPSRV-AIX

APAR status

• Closed as program error.

Error description

• The problem may occur when using STGRul and the following
  symptoms might be seen:
  -	The target replication processes are
  unkillable, and require server restart.
  -	The source
  replication processes leave behind locks on filespaces even
  after gone from Q PROC.
  -	These persist at least several days,
  and likely indefinitely. This also requires server restart to
  free the locks.
  
  -	The following messages are reported in the
  target actlog:
  ANR0422W Session 499278 for node NODE_NAME
  (11.1.111.11(11111)) refused - node name not registered.
  (SESSION: 499278)
  ANR0551E The client operation failed for
  session 499252 for node NODE_NAME on the (TDP Oracle AIX)
  operating system because of a lock conflict. (SESSION:
  499252)
  ANR9999D_3250896796
  admMatchNodeNamesEx(admnode.c:111111) Thread<755606>: Error
  1020 getting group list while matching nodes
  ANR9999D
  Thread<755606> issued message 9999
  ANR0538I A resource waiter
  has been aborted
  
  -	In the SHOW LOCKS, the thread 440 is
  holding other threads causing the lock:
  LockDesc:
  Type=17001(admin node name), NameSpace=0, SummMode=sLock,
  Key='VM_DDC1_DDCSPPROXV06_DM'
  Holder: (admutil.c:12709 Thread
  440) Tsn=0:596063250, Mode=sLock
  Waiter: (admutil.c:12709
  Thread 553258) Tsn=0:596071059, Mode=xLock
  LockDesc:
  Type=17001(admin node name), NameSpace=0, SummMode=sLock,
  Key='GDA-014-SQL'
  
  Holder: (admutil.c:12709 Thread 440)
  Tsn=0:596063250, Mode=sLock
  Waiter: (admutil.c:12709 Thread
  553253) Tsn=0:596065427, Mode=xLock
  Waiter: (admutil.c:12709
  Thread 553254) Tsn=0:596066803, Mode=sLock
  Waiter:
  (admutil.c:12709 Thread 427) Tsn=0:596066884,
  Mode=sLock
  Waiter: (admutil.c:12709 Thread 553256)
  Tsn=0:596069801, Mode=sLock
  Waiter: (admutil.c:12709 Thread
  553257) Tsn=0:596070767, Mode=sLock
  Waiter: (admutil.c:12709
  Thread 414) Tsn=0:596073149, Mode=sLock
  
  -	The SHOW THREADS
  show that the thread 440 is for MonReplCliThread:
  
  Show Thread
  : Thread 440, Parent 49: MonReplCliThread, Storage 108729008,
  AllocCnt -1462691876 HighWaterAmt 325655376
   tid=134b8,
  ptid=b31, det=1, zomb=0, join=0, result=0, sess=0, procToken=0,
  sessToken=0
   Stack trace:
   0x0900000000584940
  _cond_wait_global
   0x090000000058563c _cond_wait
  
  0x0900000000585fac pthread_cond_wait
   0x000000010000b2b4
  pkWaitConditionTracked
   0x000000010020d990 IPRA.$WaitForLock
  
  0x000000010020bbc4 tmLockTracked
   0x000000010025c2ec
  monLogStatusExt
   0x000000010024edcc StatusMonitorGridsThread
  
  0x0000000100011470 StartThread
   Awaiting cond waitP->waiting
  (0x1cce880f0), using mutex TMV->mutex (0x1116dc428), at
  tmlock.c(2539)
  

Local fix

• Restart the server to resolve the lock conflict
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * All IBM Spectrum Protect server users.                       *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See error description.                                       *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fixing level when available. This problem is currently *
  * projected to be fixed in levels 8.1.14.200,  8.1.15.100 and  *
  * 8.1.16. Note that this is subject to change at the           *
  * discretion of IBM.                                           *
  ****************************************************************
  

Problem conclusion

• Problem was fixed.  Platforms fixed:  AIX, Linux, and Windows.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels