Applying a policy on a transparent cloud tiering node

This topic provides description with an example about creating an ILM policy for tiering and then applying this policy to a transparent cloud tiering node.

After a cloud account is configured, you can apply an ILM policy file to configure a cloud storage tier. The policy configuration is done by using IBM Storage Scale standard ILM policy query language statements.

For more information on ILM policies, see Information lifecycle management for IBM Storage Scale.

You must create a policy and then apply this policy on the cloud services node for the ILM-based migration and recall to work for the cloud storage tier.

Note: Administrators must consider appropriate high and low disk utilization threshold values that are applicable in the data center environment.
A sample policy rule and the steps to apply the policy on a node are as follows:

     /* Sample policy.rules file for using Gateway functionality */
     /* Define an external pool for the off-line storage */
     define(
     exclude_list,
      (
       FALSE
       OR PATH_NAME LIKE '%/.mcstore/%'
      )
     )
    define(
    access_age,
    (DAYS(CURRENT_TIMESTAMP) - DAYS(ACCESS_TIME))
    )
    define(
    mb_allocated,
    (INTEGER(KB_ALLOCATED / 1024))
    )
    define(
    weight_expression,
    (CASE
     /*=== The file is very young, the ranking is very low ===*/
     WHEN access_age <= 1 THEN 0
     /*=== The file is very small, the ranking is low ===*/
     WHEN mb_allocated < 1 THEN access_age
     /*=== The file is resident and large and old enough,
     the ranking is standard ===*/
     ELSE mb_allocated * access_age
     END)
     )
    /* Define an external pool for the off-line storage */
    RULE EXTERNAL POOL 'mcstore' EXEC '/opt/ibm/MCStore/bin/mcstore' OPTS '-F'
    /* Define migration rule with a threshold to trigger low space events
    and move data to the external off-line pool. When on-line usage
    exceeds 25% utilization, it will move the coldest files to off-line storage
    until the on-line usage is reduced to 20% utilization level. Only files that have
    data on-line are eligible for migration. */
    RULE 'MoveOffline' MIGRATE FROM POOL 'system'
    THRESHOLD(25,20)
    WEIGHT(weight_expression)
    TO POOL 'mcstore'
    WHERE(KB_ALLOCATED > 0) AND NOT(exclude_list)
    /* Define default placement rule */
    RULE 'Placement' SET POOL 'system'
For more information on how to work with the external storage pools and related policies, see Working with external storage pools.
Note: Ensure that only a single instance of the policy is applied to migrate data to the external cloud storage pool. This avoids any potential locking issues that might arise due to multiple policy instances that try to work on the same set of files.

To ensure proper invocation of the policy on reaching threshold limits, see Threshold based migration using callbacks example.

In the sample policy, the ‘OpenRead’ & ‘OpenWrite’ rule sections represent the transparent recall of a migrated or non-resident file. transparent cloud tiering software adds its own extended attributes (dmapi.MCEA) to each file it processes. Displacement 5 in the extended attributes indicate the resident state of the file. If it is ‘N’ (non-resident), the policy issues a recall request to bring back the data from the cloud storage to the local file system for the requested Read or Write operation.

To apply a threshold policy to a file system, see Using thresholds to migrate data between pools.

IBM Storage Scale also gives administrators a way to define policies to identify the files for migration, and apply those policies immediately using the mmapplypolicy command. This is different from the threshold-based policies (which are applied by using the mmchpolicy command). The transparent cloud tiering service currently does not support parallelism in migrating files simultaneously, but parallelism in the mmapplypolicy command can be used to improve the overall throughput. Additionally, parallelism can be achieved by using an ILM policy to migrate data or by driving separate, parallel CLI commands.

A sample command to apply a policy is given here:
mmapplypolicy gpfs0 -P <rules.file> -m 24 -B 100 -g <global-work-directory> -N <tct-nodeclass>
where,
  • gpfs0 indicates the IBM Storage Scale system
  • -m indicates the number of threads created and dispatched during policy execution phase. Use the mmcloudgateway command configuration tuning settings to set your migrate or recall thread counts.
    Note: You must know the number of processors that are available on your Transparent cloud tiering service node.
  • -B indicates the maximum number of files passed to each invocation of the EXEC script specified in the <rules.file>
  • -g indicates a global work directory where IBM Storage Scale ILM policy keeps temporary data. This location/folder should be outside the folder/location being migrated. Otherwise, any temporary files that policy generates might get picked up for migration too, and migration of those temporary files might fail if those are removed by policy while they are being migrated.
  • -N indicates the transparent cloud tiering node class/nodes to which the migration workload would be distributed to further improve parallelism and in turn performance.
Note: These two parameters (-m and -B) can be adjusted to improve the performance of large-scale migrations.
The following sample policies are available in the package in the /opt/ibm/MCStore/samples folder:
Table 1. Sample policy list
No Policy Name Description
1 cloudDestroy.policy.template Apply this policy for manually destroying orphaned cloud objects before retention time expires.
2 coresidentMigrate.template Apply this policy for migrating files in the co-resident state, so that applications do not need to frequently recall files.
3 coResidenttoResident.template Apply this policy if you want to convert all "co-resident" files in a file system to "resident".
  CoresToNonres.sobar.template This is used during SOBAR restore to update the extended attributes (EAs) of co-resident files to non-resident, so that we could recall them on SOBAR restored site. Not required to be used outside SOBAR.
  exportfiles.policy.template This is used to show how to export files from a given path. similar to migrateFromDirectory.template. Can be used by customers.
4 listMigratedFiles.template This policy will list all co-resident and resident files in the file system.
5 migrateFromDirectory.policy.template This policy migrates all files in a specified directory to the cloud storage tier.
  migrateToSpecificCloudService.policy.template This is used to show how to use a particular cloud service, to migrate files via policy to a particular cloud tier. Can be used by customers.
6 recallFromCloud.policy.template Apply this policy to recall files from the cloud storage tier.
7 thresholdBasedMigration.policy.template Apply this policy to automatically migrate files from the file system to the cloud storage upon reaching certain threshold levels.
8 thumbnailTransparentRecall.policy.template This policy will help you display the thumbnails when files are listed in tools such as Windows Explorer.
9 transparentRecall.policy.template Transparent recall pulls files from the cloud when they are accessed by an application (read or write).