IT10674: LARGE NUMBER OF INCLUDE.COMPRESSION OR EXCLUDE.COMPRESSION STATEMENTS CAN SUBSTANTIALLY INCREASE TIME TO SCAN A FILE SYSTEM

IBM Spectrum Protectâ„¢ client 7.1.3.x interim fix downloads
IBM Spectrum Protectâ„¢ Client 7.1.4 Downloads and READMEs

APAR status

• Closed as program error.

Error description

• A large number of INCLUDE or EXCLUDE statements can cause
  significant increases in processing time for incremental backup
  operations. This is because as the file systems are scanned,
  each file is checked against the INCLUDE/EXCLUDE list to
  determine whether the file is eligible for backup processing.
  For a large INCLUDE/EXCLUDE list, the time needed to compare
  each file to each INCLUDE/EXCLUDE pattern can be significant:
  the larger the INCLUDE/EXCLUDE list, the more time needed to
  compare each file.
  
  Although the comparisons are necessary for INCLUDE and EXCLUDE
  entries that determine if the file can be included for backup,
  certain INCLUDE/EXCLUDE statements have no bearing on this
  determination; but they are nonetheless part of the list against
  which each file is compared. For example, these entries are not
  used to determine whether the file is to be included for, or
  excluded from, backup:
  
     INCLUDE.COMPRESSION
     EXCLUDE.COMPRESSION
     INCLUDE.SUBFILE
     EXCLUDE.SUBFILE
     INCLUDE.DEDUP
     EXCLUDE.DEDUP
     INCLUDE.ENCRYPT
     EXCLUDE.ENCRYPT
  
  The preferred behavior is for the backup-archive client to not
  use these INCLUDE/EXCLUDE patterns when determining a file's
  eligibility for backup. Not performing these comparisons would
  lead to little or no performance "penalty" for having a large
  number of entries for such INCLUDE/EXCLUDE statements.
  
  It should be observed that this APAR is not intended to address
  the case for large INCLUDE/EXCLUDE statements that are used to
  determine whether a file is included for backup.
  

Local fix

• Some possible ways to mitigate this problem:
  
     * Optimize the INCLUDE/EXCLUDE list to remove redundant or
       unnecessary patterns.
  
     * When possible, combine multiple similar patterns into
       fewer patterns. For example:
  
          EXCLUDE.COMPRESSION *.mp3
          EXCLUDE.COMPRESSION *.mp4
          EXCLUDE.COMPRESSION *.mpv
  
       Can be combined into one statement:
  
          EXCLUDE.COMPRESSION *.mp[34v]
  
     * In certain cases it might reduce elapsed processing time to
       not use EXCLUDE.COMPRESSION, but instead to use
       COMPESSALWAYS NO. While this could lead to more retries for
       files that grow during compression, the overall elapsed
       processing time might still be improved upon, expecially if
       the file systems contain very large numbers of files but
       the number of files backed up is relatively small.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * Backup-archive client versions 6.3, 6.4, and  7.1 running on *
  * all platforms.                                               *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See ERROR DESCRIPTION                                        *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fixing level when available. This problem is currently *
  * projected to be fixed                                        *
  *  in levels 7.1.4. Note that this is subject to change at the *
  * discretion of IBM.                                           *
  ****************************************************************
  

Problem conclusion

• Now lengthy EXCLUDE.COMPRESSION lists have no perceptible effect
  on the elapsed processing time when compared to not using the
  list at all.
  

Temporary fix

• A fix for this problem is currently targeted for interim fix
  packages 6.4.3.1 and 7.1.3.1. Note that until these interim
  fixes are actually available, this information is subject to
  change at the discretion of IBM.
  

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Modules/Macros

• dsmc
  

Fix information

• Fixed component name

  TSM CLIENT

• Fixed component ID

  5698ISMCL

Applicable component levels

• R64A PSY

     UP

• R64H PSY

     UP

• R64L PSY

     UP

• R64M PSY

     UP

• R64S PSY

     UP

• R64W PSY

     UP

• R71A PSY

     UP

• R71H PSY

     UP

• R71L PSY

     UP

• R71M PSY

     UP

• R71S PSY

     UP

• R71W PSY

     UP