IT33827: ANR3350W CONFIGURATION MANAGER HANGS ADMIN AND SQL QUERIES

8.1.12.000-IBM-SPSRV-WindowsX64
8.1.12.000-IBM-SPSRV-Linuxx86_64
8.1.12.000-IBM-SPSRV-Linuxs390x
8.1.12.000-IBM-SPSRV-Linuxppc64le
8.1.12.000-IBM-SPSRV-AIX
IBM Spectrum Protect Server V8.1 Fix Pack 12 (V8.1.12) Downloads

APAR status

• Closed as program error.

Error description

• During Configuration Manager refresh, to the subscribed
  servers, the server can hung on Admin or SQL query requests. In
  the reported case the administrator had to halt all of the hung
  dsmserv servers to clear all the hung queries.
  
  The key
  message to look for, on the hung server, is message:
  ANR3350W
  Locally defined domain XYZ contains at least
  one node and
  cannot be replaced with a definition from
  the configuration
  manager.
  
  The servermon data had locks on "CS Universe" and "PM
  Universe". Any admin or SQL query for schedules (CS) or policy
  (PM) will produce this server hang.
  
  Example of the servermon
  data:
  
  show lock
  LockDesc: Type=53001(cs universe),
  NameSpace=0, SummMode=ixLock, Key=''
  Holder: (csutil.c:610
  Thread 184) Tsn=0:35842, Mode=ixLock
  ...
  Waiter: (csutil.c:610
  Thread 85) Tsn=0:75812, Mode=isLock
  
  In this case, the holder
  is the Configuration Manager thread updating the subscribed
  servers:
  
  In the show TXNT, we see:
  Tsn=0:35842,
  Resurrected=False, InFlight=True, Distributed=False,
  Persistent=True, Addr 0000000001FAA240
  Start ThreadId=184,
  Timestamp=mm/dd/yyyy time, Creator=smcc.c(2145)
  Last known in
  use by ThreadId=184
  
  ...
  Type=15000(pm universe), NameSpace=0,
  SummMode=ixLock, Mode=ixLock, Key='' File=pmutil.c
  Line=1376
  
  The hung admin is a waiter:
  TXNT:
  Tsn=0:75810,
  Resurrected=False, InFlight=True, Distributed=False,
  Persistent=True, Addr 0000000014642E60
  Start ThreadId=85,
  Timestamp=mm/dd/yyyy time, Creator=admcmd.c(2635)
  Last known in
  use by ThreadId=85
  
  Spectrum Protect Versions Affected: 7.1 and
  8.1 all platforms
  
  Initial Impact: Medium
  
  Additional
  Keywords: TS003843090 Select statement
  

Local fix

• Correct the profile associated with domain that was reported in
  the message "ANR3350W" on the subscribed systems. Restart any
  servers that are hung.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * All IBM Spectrum Protect server users.                       *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See ERROR DESCRIPTION.                                       *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fixing levels when available.                          *
  * This problem is currently projected to be fixed in level     *
  * 8.1.12.                                                      *
  * Note that this is subject to change at the discretion of     *
  * IBM.                                                         *
  ****************************************************************
  

Problem conclusion

• This problem was fixed.
  Affected platforms for reported release:  AIX, HP-UX, Solaris,
  Linux, and Windows.
  Platforms fixed:  AIX, Linux, and Windows.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels

• R81A PSY

     UP

• R81L PSY

     UP

• R81W PSY

     UP