Topic
  • 5 replies
  • Latest Post - ‏2013-01-30T20:39:10Z by dlmcnabb
wettstein
wettstein
8 Posts

Pinned topic mmbackup shadow compensation

‏2012-09-07T15:10:41Z |
Hi,

I'm using mmbackup to backup to TSM. It seems to be working fine except if there are a large number of failures due to deleted files or TSM issues or whatever. There are currently around 20 million files on this file system. At the end of the backup, it does the shadow compensation and that just takes forever. Here is end of my current mmbackup session:
mmbackup: TSM Summary Information:
Total number of objects inspected: 879883
Total number of objects backed up: 614848
Total number of objects updated: 0
Total number of objects rebound: 0
Total number of objects deleted: 0
Total number of objects expired: 6601
Total number of objects failed: 131587
Fri Sep 7 07:28:38 2012 mmbackup:Total 131587 files not backed up to TSM server(s).
Fri Sep 7 07:28:38 2012 mmbackup:Analyzing audit log file /gpfs/cap1/mmbackup.audit.gpfs_cap1.tsm01
Fri Sep 7 07:28:40 2012 mmbackup:131587 files not backed up for this server. ( failed:131587 )
Fri Sep 7 07:28:40 2012 mmbackup:Worst TSM exit 4
Fri Sep 7 07:28:40 2012 mmbackup:131587 failures were logged. Compensating shadow database...
Fri Sep 7 08:00:51 2012 mmbackup:still compensating: 5869 of 131587 processed, 0 unresolved,
0 duplicates, 0 unrequested in 12 pass(es)
Fri Sep 7 08:32:54 2012 mmbackup:still compensating: 5899 of 131587 processed, 0 unresolved,
0 duplicates, 0 unrequested in 24 pass(es)
Fri Sep 7 09:05:29 2012 mmbackup:still compensating: 5922 of 131587 processed, 0 unresolved,
0 duplicates, 0 unrequested in 36 pass(es)
Fri Sep 7 09:37:24 2012 mmbackup:still compensating: 5942 of 131587 processed, 0 unresolved,
0 duplicates, 0 unrequested in 48 pass(es)
It is going to take forever to complete this. I usually just kill mmbackup because rebuilding the shadow database only takes about an extra 20 minutes at the start of the backup. What can be done to improve this? Is there an option to just skip the shadow compensation?
Updated on 2013-01-30T20:39:10Z at 2013-01-30T20:39:10Z by dlmcnabb
  • sberman
    sberman
    62 Posts

    Re: mmbackup shadow compensation

    ‏2012-09-07T17:02:24Z  
    Yes, indeed 131,000 errors will take a lot of compensation to clean up the shadow database for all these TSM failures. We are currently working on an improvement that through the use of pre-ordering the failure list will reduce the number of passes over the database that are needed to effect the compensation phase to a single pass. This will speed things up geometrically by the product of number of files in the file system times the number of errors encountered. So expect a big improvement in the next PTFs.

    In the interim, killing it and rebuilding is a fine work-around.
    -Steve from GPFS
  • wettstein
    wettstein
    8 Posts

    Re: mmbackup shadow compensation

    ‏2013-01-30T15:25:20Z  
    • sberman
    • ‏2012-09-07T17:02:24Z
    Yes, indeed 131,000 errors will take a lot of compensation to clean up the shadow database for all these TSM failures. We are currently working on an improvement that through the use of pre-ordering the failure list will reduce the number of passes over the database that are needed to effect the compensation phase to a single pass. This will speed things up geometrically by the product of number of files in the file system times the number of errors encountered. So expect a big improvement in the next PTFs.

    In the interim, killing it and rebuilding is a fine work-around.
    -Steve from GPFS
    Hi,

    Has there been any improvements in this? I've been looking over the release notes of GPFS updates and I don't think I saw anything about this. We're still running GPFS 3.4 (3.4.0-17 to be exact). Does GPFS 3.5 improve the situation?

    Thanks
    Andy
  • sberman
    sberman
    62 Posts

    Re: mmbackup shadow compensation

    ‏2013-01-30T17:28:16Z  
    • wettstein
    • ‏2013-01-30T15:25:20Z
    Hi,

    Has there been any improvements in this? I've been looking over the release notes of GPFS updates and I don't think I saw anything about this. We're still running GPFS 3.4 (3.4.0-17 to be exact). Does GPFS 3.5 improve the situation?

    Thanks
    Andy
    I happen to be working on that today :)
    I plan to try to eliminate recently deleted objects from the lists of objects created for TSM to process. Once complete and tested, this will make its way into both 3.4 and 3.5 service streams. I do not yet have an expected date for that though.

    If you have the option to move up to GPFS 3.5 TL2 we support mmbackup from a snapshot and that also will eliminate errors from recently deleted objects.
  • renarg
    renarg
    127 Posts

    Re: mmbackup shadow compensation

    ‏2013-01-30T20:01:02Z  
    • sberman
    • ‏2013-01-30T17:28:16Z
    I happen to be working on that today :)
    I plan to try to eliminate recently deleted objects from the lists of objects created for TSM to process. Once complete and tested, this will make its way into both 3.4 and 3.5 service streams. I do not yet have an expected date for that though.

    If you have the option to move up to GPFS 3.5 TL2 we support mmbackup from a snapshot and that also will eliminate errors from recently deleted objects.
    Hallo Steven,

    can you explain which version number is GPFS 3.5 TL2.

    Regards Renar
  • dlmcnabb
    dlmcnabb
    1012 Posts

    Re: mmbackup shadow compensation

    ‏2013-01-30T20:39:10Z  
    • renarg
    • ‏2013-01-30T20:01:02Z
    Hallo Steven,

    can you explain which version number is GPFS 3.5 TL2.

    Regards Renar
    Tl2 was the temporary name for all maintenance releases from 3.5.0.7 onward.