I'm using mmbackup to backup to TSM. It seems to be working fine except if there are a large number of failures due to deleted files or TSM issues or whatever. There are currently around 20 million files on this file system. At the end of the backup, it does the shadow compensation and that just takes forever. Here is end of my current mmbackup session:
mmbackup: TSM Summary Information:
Total number of objects inspected: 879883
Total number of objects backed up: 614848
Total number of objects updated: 0
Total number of objects rebound: 0
Total number of objects deleted: 0
Total number of objects expired: 6601
Total number of objects failed: 131587
Fri Sep 7 07:28:38 2012 mmbackup:Total 131587 files not backed up to TSM server(s).
Fri Sep 7 07:28:38 2012 mmbackup:Analyzing audit log file /gpfs/cap1/mmbackup.audit.gpfs_cap1.tsm01
Fri Sep 7 07:28:40 2012 mmbackup:131587 files not backed up for this server. ( failed:131587 )
Fri Sep 7 07:28:40 2012 mmbackup:Worst TSM exit 4
Fri Sep 7 07:28:40 2012 mmbackup:131587 failures were logged. Compensating shadow database...
Fri Sep 7 08:00:51 2012 mmbackup:still compensating: 5869 of 131587 processed, 0 unresolved,
0 duplicates, 0 unrequested in 12 pass(es)
Fri Sep 7 08:32:54 2012 mmbackup:still compensating: 5899 of 131587 processed, 0 unresolved,
0 duplicates, 0 unrequested in 24 pass(es)
Fri Sep 7 09:05:29 2012 mmbackup:still compensating: 5922 of 131587 processed, 0 unresolved,
0 duplicates, 0 unrequested in 36 pass(es)
Fri Sep 7 09:37:24 2012 mmbackup:still compensating: 5942 of 131587 processed, 0 unresolved,
0 duplicates, 0 unrequested in 48 pass(es)
It is going to take forever to complete this. I usually just kill mmbackup because rebuilding the shadow database only takes about an extra 20 minutes at the start of the backup. What can be done to improve this? Is there an option to just skip the shadow compensation?
sberman 27000268QN76 Posts
Re: mmbackup shadow compensation2012-09-07T17:02:24ZThis is the accepted answer. This is the accepted answer.Yes, indeed 131,000 errors will take a lot of compensation to clean up the shadow database for all these TSM failures. We are currently working on an improvement that through the use of pre-ordering the failure list will reduce the number of passes over the database that are needed to effect the compensation phase to a single pass. This will speed things up geometrically by the product of number of files in the file system times the number of errors encountered. So expect a big improvement in the next PTFs.
In the interim, killing it and rebuilding is a fine work-around.
-Steve from GPFS
wettstein 27000576X88 Posts
Re: mmbackup shadow compensation2013-01-30T15:25:20ZThis is the accepted answer. This is the accepted answer.
- sberman 27000268QN
Has there been any improvements in this? I've been looking over the release notes of GPFS updates and I don't think I saw anything about this. We're still running GPFS 3.4 (3.4.0-17 to be exact). Does GPFS 3.5 improve the situation?
sberman 27000268QN76 Posts
Re: mmbackup shadow compensation2013-01-30T17:28:16ZThis is the accepted answer. This is the accepted answer.
- wettstein 27000576X8
I plan to try to eliminate recently deleted objects from the lists of objects created for TSM to process. Once complete and tested, this will make its way into both 3.4 and 3.5 service streams. I do not yet have an expected date for that though.
If you have the option to move up to GPFS 3.5 TL2 we support mmbackup from a snapshot and that also will eliminate errors from recently deleted objects.