Cache eviction

Cache eviction is a feature where file data blocks in the cache are released when fileset usage exceeds the fileset soft quota, and space is created for new files.

The process of releasing blocks is called eviction. However, file data is not evicted if the file data is dirty. A file whose outstanding changes are not flushed to home is a dirty file.

You can use automatic cache eviction or define your own policy to decide which file data is evicted. To automatically enable cache eviction, define a fileset soft quota on the cache fileset. Eviction starts when fileset usage reaches the soft quota limit. A time lag might result when an eviction is triggered and the data is evicted. Tuning the soft and hard quota limits minimizes application delays that are caused by the data being cached at a faster rate than it is being evicted.

Cache eviction based on inode quotas is not supported. Cached data from partially fetched files are not evicted from cache. When files are evicted from cache, all data blocks of those files are cleared. The files are then uncached. When a read operation is performed on the files the next time, the files fetch data from home.

Cache eviction is enabled by default on all AFM nodes and is controlled by the afmEnableAutoEviction parameter, and fileset block quota. Cache eviction can also be manually triggered by using the mmafmctl evict command. When a file is evicted, file data is removed from the cache, but the inode stays in the cache. Using eviction, you can build environments, where all objects from home are available but running in limited amount of space.

For example, a cache could be created in flash storage. File eviction opens a powerful method of providing small but high speed and low latency cache filesets to clusters.

Manual eviction can be done by using the mmafmctl evict command as under:
mmafmctl Device evict -j FilesetName 
                 [--safe-limit SafeLimit] [--order {LRU | SIZE}]
                 [--log-file LogFile] [--filter Attribute=Value ...] 
                 [--list-file ListFile] [--file FilePath]
For more information, see mmafmctl command.

This option is applicable for RO/SW/IW/LU filesets. This command can be run manually or run in a script with a custom policy to implement a custom eviction policy. Options can be combined.

--safe-limit SafeLimit - This is a mandatory parameter for the manual evict option, for order and filter attributes. It specifies target quota limit which is used as the low water mark for eviction in bytes – the value must be less than the soft limit. This parameter can be used alone or can be combined with one of the following parameters (order or filter attributes). Specify the parameter in bytes.

--order LRU | SIZE - The order in which files are to be chosen for eviction: LRU - Least recently used files are to be evicted first. SIZE - Larger-sized files are to be evicted first.

--log-file Log File- The file where the eviction log is to be stored. By default no logs are generated.

--filter Attribute=Value - The attributes that you can use to control the way data is evicted from the cache. Valid attributes are: FILENAME=File Name - The name of a file to be evicted from the cache. This option uses an SQL-type search query. If the same file name exists in more than one directory, the cache will evict all the files with that name. The complete path to the file must not be given here. MINFILESIZE=Size - The minimum size of a file to evict from the cache. This value is compared to the number of blocks allocated to a file (KB_ALLOCATED), which might differ slightly from the file size. MAXFILESIZE=Size - The maximum size of a file to evict from the cache. This value is compared to the number of blocks allocated to a file (KB_ALLOCATED), which might differ slightly from the file size.

Possible combinations of options could be:
  • Only Safe limit
  • Safe limit + LRU
  • Safe limit + SIZE
  • Safe limit + FILENAME
  • Safe limit + MINFILESIZE
  • Safe limit + MAXFILESIZE
  • Safe limit + LRU + FILENAME
  • Safe limit + LRU + MINFILESIZE
  • Safe limit + LRU + MAXFILESIZE
  • Safe limit + SIZE + FILENAME
  • Safe limit + SIZE + MINFILESIZE
  • Safe limit + SIZE + MAXFILESIZE

--list-file ListFile - The specified file is a file containing a list of files to be evicted, one file per line. All files must have fully qualified path names. Filesystem quotas need not be specified. If the list of files to be evicted have filenames with special characters then a policy should be used to generate the listfile. This policy output should be hand-edited to remove all other entries except the filenames and can be passed to the evict command.

--file FilePath - The fully qualified name of the file to be evicted. Filesystem quotas need not be specified.

enforceFilesetQuotaOnRoot - Enable to evict files created by root. By default, files created by root are not evicted.

  1. Evicting using filter option -
    node1:/gpfs/cache/fileset_IW # mmlsfs fs1 -Q
              flag value description
              ------------------- ------------------------------------------------------------
              Q user;group;fileset Quotas accounting enabled
              none Quotas enforced
              none Default quotas enabled  
    
    node1:/gpfs/cache/fileset_IW # mmquotaon -v fs1
              mmquotaon on fs1 
    
    node1:/gpfs/cache/fileset_IW # mmlsfs fs1 -Q
              flag value description
              ------------------- ------------------------------------------------------------
              Q user;group;fileset Quotas accounting enabled
              user;group;fileset Quotas enforced
              none Default quotas enabled 
    
    node1:/gpfs/cache/fileset_IW # mmlsquota -j fileset_IW fs1
              Block Limits | File Limits
              Filesystem type KB quota limit in_doubt grace | files quota limit in_doubt grace Remarks
              fs1 FILESET 96 102400 2097152 0 none | 20 0 0 0 none 
              
    node1:/gpfs/cache/fileset_IW # mmafmctl fs1 evict -j fileset_IW --filter FILENAME='%work%'
              mmafmctl: Run mmcheckquota command before running mmafmctl with evict option
              mmafmctl: Command failed. Examine previous error messages to determine cause.  
    node1:/gpfs/cache/fileset_IW # mmcheckquota fs1
              
              fs1: Start quota check2 % complete on Thu Oct 27 09:19:41 2016
              […]
              95 % complete on Thu Oct 27 09:20:03 2016
              100 % complete on Thu Oct 27 09:20:03 2016
              Finished scanning the inodes for fs1.
              Merging results from scan. 
              
    node1:/gpfs/cache/fileset_IW # mmafmlocal ls workfile1
              -rw-r--r-- 1 root root 104857600 Oct 27 08:30 workfile1 
              
    node1:/gpfs/cache/fileset_IW # mmafmctl fs1 evict -j fileset_IW --filter FILENAME='%work%'
              node1:/gpfs/cache/fileset_IW # mmafmlocal ls workfile1
              mmafmlocal: Command failed. Examine previous error messages to determine cause.
  2. Manual evict using the --list-file option -
    [root@c21f2n08 ~]# ls -lshi /gpfs/fs1/evictCache
              total 6.0M
              27858308 1.0M -rw-r--r--. 1 root root 1.0M Feb  5  02:07 file1M
              27858307 2.0M -rw-r--r--. 1 root root 2.0M Feb  5  02:07 file2M
              27858306 3.0M -rw-r--r--. 1 root root 3.0M Feb  5  02:07 file3M 
    
    [root@c21f2n08 ~]# echo "RULE EXTERNAL LIST 'HomePREPDAEMON' RULE 'ListLargeFiles' LIST
              'HomePREPDAEMON' WHERE PATH_NAME LIKE '%'" > /tmp/evictionPolicy.pol 
    
    [root@c21f2n08 ~]# mmapplypolicy /gpfs/fs1/evictCache -I defer -P /tmp/evictionPolicy.pol -f /tmp/evictionList
               #Edited list of files to be evicted
    [root@c21f2n08 ~]# cat /tmp/evictionList.list.HomePREPDAEMON
               27858306 605742886 0   --/gpfs/fs1/evictCache/file3M 
    
    [root@c21f2n08 ~]# mmafmctl fs1 evict –j evictCache --list-file /tmp/evictionList.list.HomePREPDAEMON

    [root@c21f2n08 ~]# ls -lshi /gpfs/fs1/evictCachetotal 3.0M

                27858308 1.0M -rw-r--r--. 1 root root 1.0M Feb  5  02:07 file1M
                27858307 2.0M -rw-r--r--. 1 root root 2.0M Feb  5  02:07 file2M
                27858306    0 -rw-r--r--. 1 root root 3.0M Feb  5 02:07 file3M
  3. Manual evict using the --file option -
    [root@c21f2n08 ~]# ls -lshi /gpfs/fs1/evictCache
              total 3.0M
              27858308 1.0M -rw-r--r--. 1 root root 1.0M Feb  5  02:07 file1M
              27858307 2.0M -rw-r--r--. 1 root root 2.0M Feb  5  02:07 file2M
              27858306    0 -rw-r--r--. 1 root root 3.0M Feb  5 02:07 file3M 
              
    [root@c21f2n08 ~]# mmafmctl fs1 evict –j evictCache --file /gpfs/fs1/evictCache/file1M

    [root@c21f2n08 ~]# ls -lshi /gpfs/fs1/evictCache/file1M

             total 0
             27858308 0 -rw-r--r--. 1 root root 1.0M Feb  5 02:07 file1M