Cache eviction

Cache eviction is a feature where file data blocks in the cache are released when a fileset usage exceeds the fileset soft quota, and space is created for new files.

The process of releasing blocks is called eviction. However, file data is not evicted if the file data is dirty. A file whose outstanding changes are not flushed to home is a dirty file.

You can use automatic cache eviction or define your own policy to decide which file data needs to be evicted. To automatically enable cache eviction, define a fileset soft quota on the cache fileset. Eviction starts when fileset usage reaches the soft quota limit. A time lag might result when an eviction is triggered and the data is evicted. Tuning the soft and hard quota limits minimizes application delays because the data is being cached at a faster rate than it is being evicted.

Cache eviction based on inode quotas is not supported. Cached data from partially fetched files can be evicted manually by using --file parameter of mmafmctl command. When files are evicted from cache, all data blocks of those files are cleared. The files are then uncached. When a read operation is performed on the files the next time, the files fetch data from home.

Cache eviction is enabled by default on all AFM nodes and is controlled by the afmEnableAutoEviction parameter, and fileset block quota. Cache eviction can also be manually triggered by using the mmafmctl evict command. When a file is evicted, file data is removed from the cache, but the inode stays in the cache. Using eviction, you can build environments, where all objects from home are available but running in limited amount of space.

For example, a cache can be created in flash storage. File eviction opens a powerful method of providing small but high speed and low latency cache filesets to clusters.

Manual eviction can be done by using the mmafmctl evict command as follows:
mmafmctl Device evict -j FilesetName 
                 [--safe-limit SafeLimit] [--order {LRU | SIZE}]
                 [--log-file LogFile] [--filter Attribute=Value ...] 
                 [--list-file ListFile] [--file FilePath] [--range]

For more information, see mmafmctl command.

This option is applicable for RO/SW/IW/LU filesets. This command can be run manually or run in a script with a custom policy to implement a custom eviction policy. Options can be combined.

--safe-limit SafeLimit
This parameter is mandatory for the manual eviction, for order and filter attributes. It specifies target quota limit that is used as the low water mark for eviction in bytes – the value must be less than the soft limit. This parameter can be used alone or can be combined with one of the following parameters (order or filter attributes). Specify the parameter in bytes.
--order LRU | SIZE
The order in which files are to be chosen for eviction: LRU - Least recently used files are to be evicted first. SIZE - Larger-sized files are to be evicted first.
LRU
AFM updates the file's atime when the specified file is read from the target/home even though the atimeDeferredSeconds parameter indicates otherwise. File's atime is not updated when the already cached file is read. This is to support auto eviction based on LRU algorithm. Reading the file from the target/home affects performance. So, atime is set after reading the file from target/home to make sure that the latest file read from target/home is evicted based on LRU.
--log-file Log File
The file where the eviction log is to be stored. By default no logs are generated.
--filter Attribute=Value
The attributes that you can use to control the way data is evicted from the cache. Valid attributes are: FILENAME=File Name - The name of a file to be evicted from the cache. This option uses an SQL-type search query. If the same file name exists in more than one directory, the cache evicts all the files with that name. The complete path to the file must not be given here. MINFILESIZE=Size - The minimum size of a file to evict from the cache. This value is compared to the number of blocks that are allocated to a file (KB_ALLOCATED), which might differ slightly from the file size. MAXFILESIZE=Size - The maximum size of a file to evict from the cache. This value is compared to the number of blocks that are allocated to a file (KB_ALLOCATED), which might differ slightly from the file size.
Possible combinations of options are:
  • Only Safe limit
  • Safe limit + LRU
  • Safe limit + SIZE
  • Safe limit + FILENAME
  • Safe limit + MINFILESIZE
  • Safe limit + MAXFILESIZE
  • Safe limit + LRU + FILENAME
  • Safe limit + LRU + MINFILESIZE
  • Safe limit + LRU + MAXFILESIZE
  • Safe limit + SIZE + FILENAME
  • Safe limit + SIZE + MINFILESIZE
  • Safe limit + SIZE + MAXFILESIZE
--list-file ListFile
The specified file contains a list of files to be evicted, one file per line. All files must have fully qualified path names. File system quotas need not be specified. If the list of files to be evicted have file names with special characters, then a policy must be used to generate the listfile. This policy output must be hand-edited to remove all other entries except the file names and can be passed to the evict command.
--file FilePath

The fully qualified name of the file to be evicted. File system quotas need not be specified.

enforceFilesetQuotaOnRoot – Enable to evict files created by root. By default, the files that are created by root are not evicted.

--range
Enables eviction of a range of specified data blocks from a file. When this option is used with the --list-file option, AFM calculates the offset and length data blocks of a file that is provided in the --list-file file, and evicts only these data blocks from the file. With the range eviction, AFM provides fine granularity for eviction of specific data blocks from a file.
You can specify more than one files in the list of files. When afmEvictRange option is set on the fileset, auto-eviction and manual eviction (without range) evicts all the data blocks except first and last data blocks. By default the whole file is evicted. After this parameter enabled on an AFM fileset, a list of files contains offset and length that can be provided for eviction.
In the following example, 5G data blocks starting from 10G till 15G are evicted:
  1. Enable the range byte eviction by setting the afmEvictRange parameter.
    mmchfileset fs1 fset1 -p afmEvictRange=yes
  2. Check whether the list-file is in the following format:
    <Start_Offset> <length> <filepath>
  3. List the list-file information.
    ls -lsh /gpfs/fs1/ro1/bigfile1
    A sample output is as follows:
    19458 100G -rw-r--r-- 1 root root 100G Jul 16 23:35 /ibm/fs1/file1
  4. Copy the file.
    cat /tmp/rangefile
    A sample output is as follows:
    10G 15G /gpfs/fs1/ro1/bigfile1
  5. Evict the range of data blocks from the file.
    mmafmctl fs1 evict -j fileset --list-file /tmp/rangefile --range

AFM evicts data 5G data blocks starting from 10G till 15G, offset and length are aligned on block boundary. AFM evicts the rage of data blocks from the file as per file system block size calculation.

  1. Evicting by using the filter option:
    node1:/gpfs/cache/fileset_IW  mmlsfs fs1 -Q
          flag value description
          ------------------- ------------------------------------------------------------
          Q user;group;fileset Quotas accounting enabled
          none Quotas enforced
          none Default quotas enabled
    node1:/gpfs/cache/fileset_IW mmquotaon -v fs1
    
          mmquotaon on fs1
    node1:/gpfs/cache/fileset_IW mmlsfs fs1 -Q
    
          flag value description
          ------------------- ------------------------------------------------------------
          Q user;group;fileset Quotas accounting enabled
          user;group;fileset Quotas enforced
          none Default quotas enabled
    node1:/gpfs/cache/fileset_IW mmlsquota -j fileset_IW fs1
    
          Block Limits | File Limits
          Filesystem type KB quota limit in_doubt grace | files quota limit in_doubt grace Remarks
          fs1 FILESET 96 102400 2097152 0 none | 20 0 0 0 none
    node1:/gpfs/cache/fileset_IW mmafmctl fs1 evict -j fileset_IW --filter FILENAME='%work%'
    
          mmafmctl: Run mmcheckquota command before running mmafmctl with evict option
          mmafmctl: Command failed. Examine previous error messages to determine cause.
    node1:/gpfs/cache/fileset_IW mmcheckquota fs1
              
          fs1: Start quota check2 % complete on Thu Oct 27 09:19:41 2016
          […]
          95 % complete on Thu Oct 27 09:20:03 2016
          100 % complete on Thu Oct 27 09:20:03 2016
          Finished scanning the inodes for fs1.
          Merging results from scan.
    node1:/gpfs/cache/fileset_IW mmafmlocal ls workfile1
    
          -rw-r--r-- 1 root root 104857600 Oct 27 08:30 workfile1
    node1:/gpfs/cache/fileset_IW mmafmctl fs1 evict -j fileset_IW --filter FILENAME='%work%' 
    
          node1:/gpfs/cache/fileset_IW mmafmlocal ls workfile1
          mmafmlocal: Command failed. Examine previous error messages to determine cause.
  2. Manual eviction by using the --list-file option: 
    ls -lshi /gpfs/fs1/evictCache
          total 6.0M
          27858308 1.0M -rw-r--r--. 1 root root 1.0M Feb  5  02:07 file1M
          27858307 2.0M -rw-r--r--. 1 root root 2.0M Feb  5  02:07 file2M
          27858306 3.0M -rw-r--r--. 1 root root 3.0M Feb  5  02:07 file3M
    echo "RULE EXTERNAL LIST 'HomePREPDAEMON' RULE 'ListLargeFiles' LIST
    
          'HomePREPDAEMON' WHERE PATH_NAME LIKE '%'" > /tmp/evictionPolicy.pol
    mmapplypolicy /gpfs/fs1/evictCache -I defer -P /tmp/evictionPolicy.pol -f /tmp/evictionList 
    
           #Edited list of files to be evicted
    cat /tmp/evictionList.list.HomePREPDAEMON
    
           27858306 605742886 0   --/gpfs/fs1/evictCache/file3M
    mmafmctl fs1 evict –j evictCache --list-file /tmp/evictionList.list.HomePREPDAEMON
     
Evicted   (Total)     Failed
   1       (1)          0 
tspcacheevict: List of failures : /var/mmfs/tmp/evictionFailedList.mmafmctl.23349
mmafmctl: Command failed. Examine previous error messages to determine cause.

    ls -lshi /gpfs/fs1/evictCache

    
            total 3.0M
            27858308 1.0M -rw-r--r--. 1 root root 1.0M Feb  5  02:07 file1M
            27858307 2.0M -rw-r--r--. 1 root root 2.0M Feb  5  02:07 file2M
            27858306    0 -rw-r--r--. 1 root root 3.0M Feb  5 02:07 file3M
  3. Manual eviction by using the --file option:
    ls -lshi /gpfs/fs1/evictCache
          total 3.0M
          27858308 1.0M -rw-r--r--. 1 root root 1.0M Feb  5  02:07 file1M
          27858307 2.0M -rw-r--r--. 1 root root 2.0M Feb  5  02:07 file2M
          27858306    0 -rw-r--r--. 1 root root 3.0M Feb  5 02:07 file3M
    mmafmctl fs1 evict –j evictCache --file /gpfs/fs1/evictCache/file1M

    ls -lshi /gpfs/fs1/evictCache/file1M

    
         total 0
         27858308 0 -rw-r--r--. 1 root root 1.0M Feb  5 02:07 file1M
  4. Manual eviction of a partially cached file by using the --file option. In the following example, file1 is partially cached file of ro1 fileset. Out of 2MB, only 1MB data blocks are cached. You can use below steps to evict 1MB data blocks.

    # ls -altrish /gpfs/fs1/ro1

    
total 2.0M
27858311 1.0M -rw-r--r--. 1 root root 2.0M Feb  5  02:07 file1

    mmafmctl fs1 evict –j ro1 --file /gpfs/fs1/ro1/file1

    # ls -altrish /gpfs/fs1/ro1/file1
    
total 0
27858311 0 -rw-r--r--. 1 root root 2.0M Feb  5 02:07 file1M
Note: If a file is evicted from an AFM fileset, the snapshot file of the evicted file contains zeros.