Cache eviction

Cache eviction is a feature where file data blocks in the cache are released when a fileset usage exceeds the fileset soft quota, and space is created for new files.

The process of releasing blocks is called eviction. However, file data is not evicted if the file data is dirty. A file whose outstanding changes are not flushed to home is a dirty file.

You can use automatic cache eviction or define your own policy to decide which file data needs to be evicted. To automatically enable cache eviction, define a fileset soft quota on the cache fileset. Eviction starts when fileset usage reaches the soft quota limit. A time lag might result when an eviction is triggered and the data is evicted. Tuning the soft and hard quota limits minimizes application delays because the data is being cached at a faster rate than it is being evicted.

Cache eviction based on inode quotas is not supported. Cached data from partially fetched files can be evicted manually by using --file parameter of mmafmctl command. When files are evicted from cache, all data blocks of those files are cleared. The files are then uncached. When a read operation is performed on the files the next time, the files fetch data from home.

Cache eviction is enabled by default on all AFM nodes and is controlled by the afmEnableAutoEviction parameter, and fileset block quota. Cache eviction can also be manually triggered by using the mmafmctl evict command. When a file is evicted, file data is removed from the cache, but the inode stays in the cache. Using eviction, you can build environments, where all objects from home are available but running in limited amount of space.

For example, a cache can be created in flash storage. File eviction opens a powerful method of providing small but high speed and low latency cache filesets to clusters.

Manual eviction can be done by using the mmafmctl evict command as follows:
mmafmctl Device evict -j FilesetName 
                 [--safe-limit SafeLimit] [--order {LRU | SIZE}]
                 [--log-file LogFile] [--filter Attribute=Value ...] 
                 [--list-file ListFile] [--file FilePath]

For more information, see mmafmctl command.

This option is applicable for RO/SW/IW/LU filesets. This command can be run manually or run in a script with a custom policy to implement a custom eviction policy. Options can be combined.

--safe-limit SafeLimit
This parameter is mandatory for the manual eviction, for order and filter attributes. It specifies target quota limit that is used as the low water mark for eviction in bytes – the value must be less than the soft limit. This parameter can be used alone or can be combined with one of the following parameters (order or filter attributes). Specify the parameter in bytes.
--order LRU | SIZE
The order in which files are to be chosen for eviction: LRU - Least recently used files are to be evicted first. SIZE - Larger-sized files are to be evicted first.Start of change
LRU
AFM updates the file's atime when the specified file is read from the target/home even though the atimeDeferredSeconds parameter indicates otherwise. File's atime is not updated when the already cached file is read. This is to support auto eviction based on LRU algorithm. Reading the file from the target/home affects performance. So, atime is set after reading the file from target/home to make sure that the latest file read from target/home is evicted based on LRU.
End of change
--log-file Log File
The file where the eviction log is to be stored. By default no logs are generated.
--filter Attribute=Value
The attributes that you can use to control the way data is evicted from the cache. Valid attributes are: FILENAME=File Name - The name of a file to be evicted from the cache. This option uses an SQL-type search query. If the same file name exists in more than one directory, the cache evicts all the files with that name. The complete path to the file must not be given here. MINFILESIZE=Size - The minimum size of a file to evict from the cache. This value is compared to the number of blocks that are allocated to a file (KB_ALLOCATED), which might differ slightly from the file size. MAXFILESIZE=Size - The maximum size of a file to evict from the cache. This value is compared to the number of blocks that are allocated to a file (KB_ALLOCATED), which might differ slightly from the file size.
Possible combinations of options are:
  • Only Safe limit
  • Safe limit + LRU
  • Safe limit + SIZE
  • Safe limit + FILENAME
  • Safe limit + MINFILESIZE
  • Safe limit + MAXFILESIZE
  • Safe limit + LRU + FILENAME
  • Safe limit + LRU + MINFILESIZE
  • Safe limit + LRU + MAXFILESIZE
  • Safe limit + SIZE + FILENAME
  • Safe limit + SIZE + MINFILESIZE
  • Safe limit + SIZE + MAXFILESIZE

--list-file ListFile – The specified file contains a list of files to be evicted, one file per line. All files must have fully qualified path names. File system quotas need not be specified. If the list of files to be evicted have file names with special characters, then a policy must be used to generate the listfile. This policy output must be hand-edited to remove all other entries except the file names and can be passed to the evict command.

--file FilePath – The fully qualified name of the file to be evicted. File system quotas need not be specified.

enforceFilesetQuotaOnRoot – Enable to evict files created by root. By default, the files that are created by root are not evicted.

  1. Evicting by using the filter option:
    node1:/gpfs/cache/fileset_IW # mmlsfs fs1 -Q
          flag value description
          ------------------- ------------------------------------------------------------
          Q user;group;fileset Quotas accounting enabled
          none Quotas enforced
          none Default quotas enabled
    node1:/gpfs/cache/fileset_IW # mmquotaon -v fs1
    
          mmquotaon on fs1
    node1:/gpfs/cache/fileset_IW # mmlsfs fs1 -Q
    
          flag value description
          ------------------- ------------------------------------------------------------
          Q user;group;fileset Quotas accounting enabled
          user;group;fileset Quotas enforced
          none Default quotas enabled
    node1:/gpfs/cache/fileset_IW # mmlsquota -j fileset_IW fs1
    
          Block Limits | File Limits
          Filesystem type KB quota limit in_doubt grace | files quota limit in_doubt grace Remarks
          fs1 FILESET 96 102400 2097152 0 none | 20 0 0 0 none
    node1:/gpfs/cache/fileset_IW # mmafmctl fs1 evict -j fileset_IW --filter FILENAME='%work%'
    
          mmafmctl: Run mmcheckquota command before running mmafmctl with evict option
          mmafmctl: Command failed. Examine previous error messages to determine cause.
    node1:/gpfs/cache/fileset_IW # mmcheckquota fs1
              
          fs1: Start quota check2 % complete on Thu Oct 27 09:19:41 2016
          […]
          95 % complete on Thu Oct 27 09:20:03 2016
          100 % complete on Thu Oct 27 09:20:03 2016
          Finished scanning the inodes for fs1.
          Merging results from scan.
    node1:/gpfs/cache/fileset_IW # mmafmlocal ls workfile1
    
          -rw-r--r-- 1 root root 104857600 Oct 27 08:30 workfile1
    node1:/gpfs/cache/fileset_IW # mmafmctl fs1 evict -j fileset_IW --filter FILENAME='%work%' 
    
          node1:/gpfs/cache/fileset_IW # mmafmlocal ls workfile1
          mmafmlocal: Command failed. Examine previous error messages to determine cause.
  2. Manual eviction by using the --list-file option: 
    [root@c21f2n08 ~]# ls -lshi /gpfs/fs1/evictCache
          total 6.0M
          27858308 1.0M -rw-r--r--. 1 root root 1.0M Feb  5  02:07 file1M
          27858307 2.0M -rw-r--r--. 1 root root 2.0M Feb  5  02:07 file2M
          27858306 3.0M -rw-r--r--. 1 root root 3.0M Feb  5  02:07 file3M
    [root@c21f2n08 ~]# echo "RULE EXTERNAL LIST 'HomePREPDAEMON' RULE 'ListLargeFiles' LIST
    
          'HomePREPDAEMON' WHERE PATH_NAME LIKE '%'" > /tmp/evictionPolicy.pol
    [root@c21f2n08 ~]# mmapplypolicy /gpfs/fs1/evictCache -I defer -P /tmp/evictionPolicy.pol -f /tmp/evictionList 
    
           #Edited list of files to be evicted
    [root@c21f2n08 ~]# cat /tmp/evictionList.list.HomePREPDAEMON
    
           27858306 605742886 0   --/gpfs/fs1/evictCache/file3M
    [root@c21f2n08 ~]# mmafmctl fs1 evict –j evictCache --list-file /tmp/evictionList.list.HomePREPDAEMON
     
Evicted   (Total)     Failed
   1       (1)          0 
tspcacheevict: List of failures : /var/mmfs/tmp/evictionFailedList.mmafmctl.23349
mmafmctl: Command failed. Examine previous error messages to determine cause.

    [root@c21f2n08 ~]# ls -lshi /gpfs/fs1/evictCache

    
            total 3.0M
            27858308 1.0M -rw-r--r--. 1 root root 1.0M Feb  5  02:07 file1M
            27858307 2.0M -rw-r--r--. 1 root root 2.0M Feb  5  02:07 file2M
            27858306    0 -rw-r--r--. 1 root root 3.0M Feb  5 02:07 file3M
  3. Manual eviction by using the --file option:
    [root@c21f2n08 ~]# ls -lshi /gpfs/fs1/evictCache
          total 3.0M
          27858308 1.0M -rw-r--r--. 1 root root 1.0M Feb  5  02:07 file1M
          27858307 2.0M -rw-r--r--. 1 root root 2.0M Feb  5  02:07 file2M
          27858306    0 -rw-r--r--. 1 root root 3.0M Feb  5 02:07 file3M
    [root@c21f2n08 ~]# mmafmctl fs1 evict –j evictCache --file /gpfs/fs1/evictCache/file1M

    [root@c21f2n08 ~]# ls -lshi /gpfs/fs1/evictCache/file1M

    
         total 0
         27858308 0 -rw-r--r--. 1 root root 1.0M Feb  5 02:07 file1M
  4. Manual eviction of a partially cached file by using the --file option. In the following example, file1 is partially cached file of ro1 fileset. Out of 2MB, only 1MB data blocks are cached. You can use below steps to evict 1MB data blocks.

    # ls -altrish /gpfs/fs1/ro1

    
total 2.0M
27858311 1.0M -rw-r--r--. 1 root root 2.0M Feb  5  02:07 file1

    # mmafmctl fs1 evict –j ro1 --file /gpfs/fs1/ro1/file1

    # ls -altrish /gpfs/fs1/ro1/file1
    
total 0
27858311 0 -rw-r--r--. 1 root root 2.0M Feb  5 02:07 file1M
Note: If a file is evicted from an AFM fileset, the snapshot file of the evicted file contains zeros.