AFM to cloud object storage policy-based upload for manual updates mode

AFM to cloud object storage now supports policy-based upload for manual updates (MU) mode.

A policy can be defined by system administrators and run by using the mmafmcosctl reconcile command that helps automatic selection and upload of the files or objects to cloud object storage buckets. Earlier, this task was done with manual intervention by using –object-list option from the upload command.

A policy rule is an SQL-like statement that directs mmafmcosctl reconcile command to upload data to the cloud object storage based on criteria defined in the policy file.

Note: Based on policy, the mmafmcosctl reconcile command does not upload a directory with objects that do not meet the policy criteria to cloud objects storage.

A policy rule specifies one or more conditions. When the conditions are true, the specific rule applies. Conditions can be specified by SQL expressions, which can include SQL functions, variables, and file attributes.

Few available file attributes are shown in the following list:
  • File name or extension
  • File size
  • User ID and group ID
  • Date and time when the file was last accessed
  • Date and time when the file was last modified

Steps to policy-based upload for the manual updates mode

  1. According to the data created in the manual updates fileset, system administrator defines a policy. For example, upload all the objects or files with the specific names. For more information, see Creating a policy.
  2. As a system administrator, you can either install this policy permanently by using mmafmcosctl –add-policy command or run it when needed, by using mmafmcosctl –policy command.
  3. When the policy is installed permanently by using –add-policy, mmafmcosctl reconcile command uses this policy to run when mafmcosctl Device FilesetName path reconcile command is run without any option.
  4. As a system administrator, you can remove the installed policy by using mmafmcosctl Device FilesetName Path reconcile –remove-policy command.
  5. As a system administrator, you can also run a policy right away by using mmafmcosctl Device FilesetName Path reconcile –policy command. Here, the policy is not stored internally.
  6. As a system administrator, you can view the installed policy by using --list-policy option.
Note: Reconcile command also performs cleanup of files when it is run along with the policy based uploads. Files that are deleted from manual update mode fileset are queued for deletes on the Cloud Object Storage and their inodes are reclaimed.

Policy creation

As a system administrator you can define a policy for variety of matching options, here is the policy that uploads all files and directories with matching name “IBM®”.
Important: When policy is created, make sure that the LIST option must have prerequisite names that are, "dirtyFiles" for files and "dirtyDirs" for directories.

Example

Manual updates mode fileset 
Node1] mmlsfileset fs1 mufileset --afm -L

Filesets in file system 'fs1':

Attributes for fileset mufileset:
==================================
Status                                  Linked
Path                                    /gpfs/fs1/mufileset
Id                                      125
Root inode                              102236163
Parent Id                               0
Created                                 Wed Apr 20 03:54:03 2022
Comment                                 
Inode space                             99      
Maximum number of inodes                100352
Allocated inodes                        100352
Permission change flag                  chmodAndSetacl
afm-associated                          Yes
Permission inherit flag                 inheritAclOnly
Target                                  https://s3.amazonaws.com:443/mufileset
Mode                                    manual-updates
File Lookup Refresh Interval            120
File Open Refresh Interval              120
Dir Lookup Refresh Interval             120
Dir Open Refresh Interval               120
Async Delay                             disable
Last pSnapId                            0
Display Home Snapshots                  no
Parallel Read Chunk Size                0
Number of Gateway Flush Threads         8
Prefetch Threshold                      0 (default)
Eviction Enabled                        yes (default)
IO Flags                                0x8280000 (afmObjectXattr,afmObjectDirectoryObj,afmObjectACL)
IO Flags2                               0x0 
Node1]
Files present in the manual-updates mode fileset 
Node1]  ls -l /gpfs/fs1/mufileset/
total 0
-rw-r--r-- 1 root root 0 Apr 29 08:42 extrafile1
-rw-r--r-- 1 root root 0 Apr 29 08:42 extrafile2
-rw-r--r-- 1 root root 0 Apr 29 08:42 extrafile3
-rw-r--r-- 1 root root 0 Apr 29 08:42 extrafile4
-rw-r--r-- 1 root root 0 Apr 29 08:42 extrafile5
-rw-r--r-- 1 root root 0 Apr 29 08:41 file1
-rw-r--r-- 1 root root 0 Apr 29 08:41 file2
-rw-r--r-- 1 root root 0 Apr 29 08:41 file3
-rw-r--r-- 1 root root 0 Apr 29 08:41 file4
-rw-r--r-- 1 root root 0 Apr 29 08:41 file5
-rw-r--r-- 1 root root 0 Apr 29 08:42 IBM1
-rw-r--r-- 1 root root 0 Apr 29 08:42 IBM2
-rw-r--r-- 1 root root 0 Apr 29 08:42 oneIBM
-rw-r--r-- 1 root root 0 Apr 29 08:42 twoIBM

Policy to transfer files which have IBM in their names respectively 
Node1] cat policyfile1
RULE EXTERNAL LIST 'dirtyFiles'
RULE 'dirtyFilesRule' LIST 'dirtyFiles'  
WHERE NAME LIKE '%IBM%' AND 
PATH_NAME NOT LIKE '$filesetPath/.pconflicts/%' AND 
PATH_NAME NOT LIKE '$filesetPath/.afm/%' AND 
PATH_NAME NOT LIKE '$filesetPath/.ptrash/%'
Issue the following command on Node1]:
mmafmcosctl fs1 mufileset /gpfs/fs1/mufileset reconcile --policy policyfile1 

Dirty file list : /var/mmfs/afm/fs1-125/recovery/policylist.data.list.dirtyFiles
     Queued    (Total)      Failed                TotalData
                                          (approx in Bytes)
         4         (4)          0                        0 

Object Upload successfully queued at the gateway.
Could not find any deleted files.
Node1] mmafmctl fs1 getstate

Fileset Name    Fileset Target                                Cache State          Gateway Node    Queue Length   Queue numExec 
------------    --------------                                -------------        ------------    ------------   ------------- 
mufileset       https://s3.amazonaws.com:443/mufileset        Active               c7f2n04         4              691
File match in name as IBM are transferred to cloud objects (seen by using cloud object shell):

[2022-04-29 08:45:37 EDT]     0B STANDARD IBM1
[2022-04-29 08:45:37 EDT]     0B STANDARD IBM2
[2022-04-29 08:45:37 EDT]     0B STANDARD oneIBM
[2022-04-29 08:45:37 EDT]     0B STANDARD twoIBM
Node1]