IT41066: CANCEL OF A REPLICATION PROCESS MAY CAUSE LOCK CONFLICT AT NEXT BACKUP OR EXPIRE INVENTORY WITH ANR0551E, ANR0918E OR ANR1894E

8.1.15.100-IBM-SPSRV-WindowsX64
8.1.15.100-IBM-SPSRV-Linuxx86_64
8.1.15.100-IBM-SPSRV-Linuxs390x
8.1.15.100-IBM-SPSRV-Linuxppc64le
8.1.15.100-IBM-SPSRV-AIX
8.1.16.000-IBM-SPCMS-WindowsX64
8.1.16.000-IBM-SPCMS-WindowsI32
8.1.16.000-IBM-SPCMS-Linuxx86_64
8.1.16.000-IBM-SPOC-WindowsX64
8.1.16.000-IBM-SPOC-Linuxx86_64
8.1.16.000-IBM-SPOC-Linuxs390x
8.1.16.000-IBM-SPOC-LinuxPPC64le
8.1.16.000-IBM-SPOC-AIX
8.1.16.000-IBM-SPSRV-WindowsX64
8.1.16.000-IBM-SPSRV-Linuxx86_64
8.1.16.000-IBM-SPSRV-Linuxs390x
8.1.16.000-IBM-SPSRV-Linuxppc64le
8.1.16.000-IBM-SPSRV-AIX
8.1.14.200-IBM-SPSRV-WindowsX64
8.1.14.200-IBM-SPSRV-Linuxx86_64
8.1.14.200-IBM-SPSRV-Linuxs390x
8.1.14.200-IBM-SPSRV-Linuxppc64le
8.1.14.200-IBM-SPSRV-AIX

APAR status

• Closed as program error.

Error description

• Stopping a REPLICATE NODE or REPLICATION STGRULE with either
  CANCEL REPLICATION / CANCEL PROCESS / or reaching DURATION LIMIT
  of the process may leave some orphan transactions alive in the
  background.
  
  This can cause different types of lock conflicts later
  depending on what operation took place for the node affected by
  the canceled replication.
  This message shows that one transaction reaches the resource
  timeout:
  
   ANR0538I A resource waiter has been aborted
  
  Next, this type of error may happen if the operation affected is
  a backup or archive:
  
   ANR0551E The client operation failed for session xx for node
   NODENAME on the (client platform) operating system because of a
   lock conflict.
  
      and/or
  
  ANR0918E Inventory Query Backup for node NODENAME  terminated -
   lock conflict.
  
  
  This error may happen if the operation affected is an EXPIRE
  INVENTORY:
  
   ANR1894E EXPIRE INVENTORY: Processing for node NODENAME, file
   space xxxx failed because of a lock conflict.
  
  
  In servermon data collection, the SHOW LOCK output shows that
  a backup or EXPIRE INVENTORY thread is waiting for a lock held
  by another transaction, but the thread owning that transaction
  does not exist anymore in the SHOW THREADS output.
  
  Example:
  
  . In below SHOW LOCK output, the lock is owned by threadid 449
  
  
  
   LockDesc: Type=19002(im node filespace), NameSpace=0,
   SummMode=sLock, Key='26.39'
    Holder: (nrfs.c:8788 Thread 449) Tsn=0:31546284, Mode=sLock
    Waiter: (imfs.c:4313 Thread 488) Tsn=0:43431356, Mode=xixLock
  
  
  . The SHOW TXNT shows the transaction and the owning Threadid
  
  
  Tsn=0:31546284, Resurrected=False, InFlight=True,
  Distributed=False, Persistent=True, Addr 000001F907B674D0
   Start ThreadId=449, Timestamp=05/14/2022 18:03:47,
  Creator=nrfs.c(8510)
  Last known in use by ThreadId=449
   Participants=3, summaryVote=ReadOnly
   EndInFlight False, endThreadId 0, tmidx 0, processBatchCount 0,
  mustAbort False.
     Participant DB: voteReceived=False, ackReceived=False
       DB: Txn 000001F933DEA580, ReadOnly(YES),
  connP=000001F843AC1230, applHandle=36750, openTbls=11:
       DB: --> OpenP=000001F8389C3D50 for table=DF.Damaged.
  
                  ...
  . But in SHOW THREADS , the threaded 449 is not found.
  
  
  . Older  SHOW THREADS output confirm that this threadid belong
    to the canceled replication process:
  
  
  ===============================================================
  Thread 449, ID 22824 (0x5928): NrReplicateFilespace,
  procToken=5, sessToken=133
  Parent=414, result=0, joining=0, detached=0, zombie=0,
  session=329
  
  Stack trace:
  78a667d4 Unknown
  7ffd75418bef WaitForMultipleObjectsEx()+ef
  7ffd75418aee WaitForMultipleObjects()+e
  7ffd45d41b31 pkWaitConditionTracked()+a1
  7ffd46a95952 EnqueueVarQueue()+102
  7ffd46a78412 ProdConsPutWork()+142
  7ffd462efa73 IterateObjectsByObjid()+3813
  7ffd462e0d93 NrReplicateFilespace()+1053
  7ffd46a7b0ba PcConsumerThread()+16a
  7ffd462de054 NrReplicateFilespaceStackTrackEntry()+14
  7ffd45d3d253 startThread()+5b3
  7ffd74effb80 o__realloc_base()+60
  7ffd761f84d4 BaseThreadInitThunk()+14
  7ffd78a11791 RtlUserThreadStart()+21
  
  Thread context:
  
  COMMAND: REPLICATE NODE
  COMMMETHOD: SSL
  PORT: 1.2.3.4:1555
  SESSION: 329
  PROCESS_NUMBER: 5
  PROCESS_DESC: Replicate Node
  THREAD_TYPE: PROCESS
  SESSION_TYPE: ADMIN
  ADMIN_NAME: XXXXX
  ===============================================================
  ===========
  
  | MDVREGR 8.1.12.0-TIV_5698MSV |
  
  IBM Spectrum Protect Versions Affected: Server  versions 8.1.12
    and above on all supported platforms
  
  Additional Keywords: TS009360342 NrReplicateFilespace
  

Local fix

• Restart the server
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * All IBM Spectrum Protect server users.                       *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See error description.                                       *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fixing level when available. This problem is currently *
  * projected to be fixed in levels 8.1.14.200 and 8.1.15.100    *
  * and 8.1.16. Note that this is subject to change at the       *
  * discretion of IBM.                                           *
  ****************************************************************
  

Problem conclusion

• This problem was fixed.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels