Manual migration and premigration with the mmapplypolicy command

Edit online

How to manually start file migration (or premigration) with an IBM Storage Scale policy file for automatic file selection.

Use the mmapplypolicy command to manually migrate or premigrate files on a GPFS file system that is managed by IBM Storage Archive Enterprise Edition. Used with a policy rule for migration or premigration, this command runs a policy that selects files according to certain criteria, and then passes these files to IBM Storage Archive EE for migration or premigration. As with automated GPFS policy-driven migrations, the name of the target IBM Storage Archive EE tape storage pool is provided as the first option of the pool definition rule in the GPFS policy file.

Note:

By default, a file must be older than 2 minutes before it can be migrated, and its mtime timestamp cannot be set to a future date. If you try to migrate the file sooner, the migration fails. You can retry the migration operation later.

For more information about the mmapplypolicy command, see the IBM Storage Scale documentation.

The following example shows an mmapplypolicy command that applies the policy according to the migration policy file migration_policyfile.txt. A migration policy file example is shown in section Example migration policy.

        mmapplypolicy <GPFS file system> -P migration_policyfile.txt -B 10000 -m 2 --single-instance

You can apply a manually created policy by manually running the mmapplypolicy command, or by scheduling the policy with the system scheduler. You can have multiple different policies, which can each include one or more rules. However, only one policy can be run at a time. To ensure that only one policy is run at a time, use the mmapplypolicy command with the --single-instance option, as shown in the preceding command example.

Important: Ensure that anIBM Storage Scale policy rule is in place to prevent migration of any IBM Storage Archive EE work directories. You must have a rule to exclude the .SpaceMan directory. If you store the IBM Storage Archive EE metadata directory in the same file system that is space that is managed by the IBM Storage Archive Enterprise Edition, you must also have a rule to exclude the IBM Storage Archive EE metadata directory. The name of the IBM Storage Archive EE metadata directory is .ltfsee. For example, if the file system name is /ibm/gpfs the metadata directory is /ibm/gpfs/.ltfsee. See Example IBM Storage Scale policy file for policy-driven migration for an example.

For information about migration performance dependencies and the IBM Storage Scale mmapplypolicy command, see Migration performance and the mmapplypolicy command.

Creating premigration rules with the THRESHOLD clause

To do manual premigration with the mmapplypolicy command, you must determine the files that need to be premigrated with the THRESHOLD clause in a GPFS policy migration rule. IBM Storage Scale does not have a premigration command, and the default behavior is to not premigrate files.

Note: The THRESHOLD clause can have the following parameters to control premigration and migration: THRESHOLD (high percentage, low percentage, premigrate percentage). If no premigrate threshold is set with the THRESHOLD clause, or a value is set greater than or equal to the low threshold, then the mmapplypolicy command does not premigrate files. If the premigrate threshold is set to zero, the mmapplypolicy command premigrates all files.

For example, the following rule premigrates all files if the storage pool occupancy lies between 0% and 30%. When the storage pool occupancy is 30% or higher, files are migrated until the storage pool occupancy drops below 30%. Then, it continues by premigrating all files.

RULE 'premig1' MIGRATE FROM POOL 'system' THRESHOLD (0,30,0) TO POOL 'ltfs'

The rule in the following example takes effect when the storage pool occupancy is higher than 50%. Then, it migrates files until the storage pool occupancy is lower than 30% after which it premigrates files.

RULE 'premig2' MIGRATE FROM POOL 'system' THRESHOLD (50,30,0) TO POOL 'ltfs'

The premigration rule can also be combined with selective file migration, as illustrated in the following example. This rule premigrates all files larger than 1 MB if the storage pool occupancy is below 30%. Otherwise, when the storage pool occupancy is 30% or higher, files are migrated until it drops below 30%: Then, it continues by premigrating all files larger than 1 MB.

RULE 'premig3' MIGRATE FROM POOL 'system' THRESHOLD (0,30,0) TO POOL 'ltfs' WHERE(  AND (KB_ALLOCATED > 5120))

The policy is configured so that if the storage pool occupancy is below 30%, it selects all files that are larger than 1 MB for premigration. Otherwise, when the storage pool occupancy is 30% or higher, the policy migrates files that are larger than 1 MB until the storage pool occupancy drops below 30%. Then, it continues by premigrating all files that are larger than 1 MB.

Example migration policy

For the purposes of this example, a file that is named migration_policyfile.txt contains the following manually created migration policy. The policy is configured to select nonzero size files that were not modified within the last two days for migration to the IBM Storage Archive EE pool named Tapepool2, which is in the tape library named library1.

Start of change from messages The example policy excludes some system files, excludes the LTFS EE metadata directory (/ibm/gpfs/.ltfsee), and files that have more than one hard links.

The SIZE parameter specifies the total amount of file size, in kilobytes, that are allowed in a single migration bucket.


/* migrate all files from the given file sytem and directory older than AGE*/
    define(
      exclude_list,
       (
         PATH_NAME LIKE '%/.SpaceMan/%' 
         OR PATH_NAME LIKE '%/.ctdb/%'  
         OR PATH_NAME LIKE '/ibm/gpfs/.ltfsee/%'
         OR NAME LIKE 'user.quota%'     
         OR NAME LIKE 'fileset.quota%'  
         OR NAME LIKE 'group.quota%'
         OR ((CURRENT_TIMESTAMP - MODIFICATION_TIME) < INTERVAL '121' SECONDS)
         OR NLINK != 1
        ) 
     )

     RULE EXTERNAL POOL 'ltfsee' 
     EXEC '/opt/ibm/ltfsee/bin/eeadm' 
     OPTS '-p Tapepool2@library1' 
     SIZE 10485760

     RULE 'ee_agemig' MIGRATE TO POOL 'ltfsee' WHERE 
     (
       LOWER(PATH_NAME) LIKE '/ibm/gpfs0/%' 
       AND (KB_ALLOCATED > 5120)  						
       AND (DAYS(CURRENT_TIMESTAMP) - DAYS(MODIFICATION_TIME) > 2) 
       AND NOT (exclude_list) 
     )

Example premigration policy

For the purposes of this example, assume that a file that is named premigration_policyfile.txt contains the following manually created premigration policy. The policy is configured so that if the storage pool occupancy is below 30%, it selects all files that are larger than 1 MB for premigration. Otherwise, when the storage pool occupancy is 30% or higher, the policy migrates files that are larger than 1 MB until the storage pool occupancy drops below 30%. Then, it continues by premigrating all files that are larger than 1 MB.

The policy is configured to premigrate the files that are larger than 1 MB to the IBM Storage Archive EE pool named Tapepool2, which is in the tape library named library1. In addition, the example policy excludes some system files, excludes the LTFS EE metadata directory (/ibm/gpfs/.ltfsee), and files that have more than one hard links.


/* Premigrate all files from the given file system and directory */
/* Define an exclude list to exclude SpaceMan, the LTFS EE metadata directory, and some other system files      */
    define(
      exclude_list,
       (
         PATH_NAME LIKE '%/.SpaceMan/%' 
         OR PATH_NAME LIKE '%/.ctdb/%'  
         OR PATH_NAME LIKE '/ibm/gpfs/.ltfsee/%'
         OR NAME LIKE 'user.quota%'     
         OR NAME LIKE 'fileset.quota%'  
         OR NAME LIKE 'group.quota%'
         OR NLINK != 1

        ) 
     )

/* Define LTFS as external pool */
     RULE EXTERNAL POOL 'ltfsee' 
     EXEC '/opt/ibm/ltfsee/bin/eeadm'    /* The full path to the eeadm command must be specified */
     OPTS '-p Tapepool2@library1'    /* 'Tapepool2' is our pool, which is located in 'library1' */

/* The following RULE is the premigration rule whereby the FILESYSTEM is given by the runpolicy script. */
/* It premigrates as long as the pool occupancy is  less than  30%                                      */
     RULE 'ee-all-premig' MIGRATE THRESHOLD (0,30,0) TO POOL 'ltfsee' WHERE
     (
       (
         PATH_NAME LIKE '/ibm/gpfs0/%'
         AND (KB_ALLOCATED > 5120)
         AND NOT (exclude_list)
        ) 
     )