IT16004: CLIENT CAN HANG DURING SCHEDULED BACKUP WHEN PROCESSING FILE SPACE WITH MILLIONS OF OBJECTS (FILES AND DIRECTORIES)

IBM Spectrum Protectâ„¢ Client 8.1.0 Downloads and READMEs
IBM Spectrum Protect client 7.1.6.x interim fix downloads

APAR status

• Closed as fixed if next.

Error description

• A client scheduler may hang in the middle of the backup
  operation and although the backup stops, all file spaces are
  backed up, the client does not report anything in either the
  error or scheduler log, the completion status is missing in the
  schedule log and is not reported to the server.
  
  Example schedule log:
  
  02-03-2016 09:10:34 --- SCHEDULEREC QUERY BEGIN
  02-03-2016 09:10:34 --- SCHEDULEREC QUERY END
  02-03-2016 09:10:34 Next operation scheduled:
  02-03-2016 09:10:34
  ------------------------------------------------------------
  02-03-2016 09:10:34 Schedule Name:         BACKUP_SCHEDULE
  02-03-2016 09:10:34 Action:                Incremental
  02-03-2016 09:10:34 Objects:
  02-03-2016 09:10:34 Options:
  02-03-2016 09:10:34 Server Window Start:   17:00:00 on
  02-03-2016
  02-03-2016 09:10:34
  ------------------------------------------------------------
  02-03-2016 09:10:34 Command will be executed in 9 hours and 12
  minutes.
  02-03-2016 18:22:34
  Executing scheduled command now.
  02-03-2016 18:22:38 Node Name: NODE1
  02-03-2016 18:22:38 Session established with server TSMSERVER:
  AIX
  02-03-2016 18:22:38   Server Version 6, Release 3, Level 5.100
  02-03-2016 18:22:38   Server date/time: 02-03-2016 18:22:38
  Last access: 02-03-2016 09:10:34
  
  02-03-2016 18:22:38 --- SCHEDULEREC OBJECT BEGIN BACKUP_SCHEDULE
  02-03-2016 17:00:00
  02-03-2016 18:22:38 Incremental backup of volume
  '\\clientnode\d$'
  02-03-2016 18:22:38 Incremental backup of volume
  '\\clientnode\d$\data1\data2'
  02-03-2016 18:22:38 Incremental backup of volume
  '\\clientnode\d$\data1\data2\data3'
  02-03-2016 18:22:38 Incremental backup of volume
  '\\clientnode\d$\data1\data3'
  02-03-2016 18:22:38 Incremental backup of volume
  '\\clientnode\d$\¸data1\data2\data3\data4'
  02-03-2016 18:22:38 Incremental backup of volume
  '\\clientnode\d$\data1\data2\data3\data4\data5'
  02-03-2016 18:22:38 Incremental backup of volume
  '\\clientnode\d$\data1\data2\data3\data4\data5\data6'
  ...
  02-03-2016 21:54:34 ANS1802E Incremental backup of
  '\\clientnode\d$\data1\data2' finished with 11 failure(s)
  02-03-2016 23:10:33 ANS1802E Incremental backup of
  '\\clientnode\d$\data1\data3' finished with 7 failure(s)
  
  
  The symptoms for this issue are:
  
  - "ANS0361I DIAG Maximum number of threads exceeded." message in
  the dsmerror.log
  
  - Hang of the backup, although all file spaces are backed up,
  however the final statistics is missing from the schedule log
  
  - A number of session with server exceeds the number of sessions
  limited by RESOUCRECEUTILIZATION client option
  
  For example, RESOUCRECEUTILIZATION 10 permits up to eight
  sessions with the server, but in the server activity log we
  could observe many more live session with the server than the
  limit of 8 in a short time span.
  
  14-04-2016 20:41:23      ANR0406I Session 311858 started for
  node NODE1 (WinNT)                           (Tcp/Ip
  10.10.10.10(55913)). (SESSION: 311858)
  
  In this case, the ANR0406I message was repeated 53 times at
  timestamp 20:41:23.
  
  Versions Affected:
  Tivoli Storage Manager client on all platforms
  
  Initial Impact:
  Medium
  
  Additional Keywords:
  TSM; Spectrum Protect; client; backup; hang; ANS1802E
  

Local fix

• Split the backup into smaller parts. A client option
  Virtualmountpoint can be used for that purpose.
  http://www.ibm.com/support/knowledgecenter/SSGSG7_7.1.3/client/r
  _opt_virtualmountpoint.html
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * Tivoli Storage Manager backup-archive client versions 6.3,   *
  * 6.4 and 7.1 running on all platforms.                        *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See ERROR DESCRIPTION                                        *
  ****************************************************************
  * RECOMMENDATION:                                              *
  ****************************************************************
  

Problem conclusion

Temporary fix

Comments

• If there is a next release of Tivoli Storage Manager after 7.1,
  this APAR will be fixed in that next release
  

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

Applicable component levels

• R71W PSN

     UP

• R71A PSN

     UP

• R71L PSN

     UP

• R71H PSN

     UP

• R71M PSN

     UP

• R71S PSN

     UP