IBM Spectrum Protect HSM for Windows Threshold Migration: Additional Information

Answer

IBM Spectum Protect HSM for Windows provides hierarchical storage management for Microsoft Windows file systems. Threshold Migration is a method to periodically and automatically create free space on the file system. You can configure the following file system policies:

• High and low thresholds determine when migration begins and ends. Migration begins when disk usage reaches the high threshold, and ends when enough space is free to meet the low threshold. Thresholds are configured as a percentage of disk usage.
• The age and size of files determine which files are candidates for threshold migration. You can configure the minimum age and size of migration candidates. You can also configure whether you want age or size to have more weight when determining which candidates to migrate first.
• You can determine how often the HSM for Windows client scans the file system for migration candidates.

The complete list of threshold migration configuration parameters and other information are described in the IBM Knowledge Center for IBM Spectrum Protect HSM for Windows, 

https://www.ibm.com/support/knowledgecenter/SSERFH_8.1.0/hsmwin/welcome…

https://www.ibm.com/support/knowledgecenter/SSERFH_8.1.0/hsmwin/c_thres…

The hsmmonitor service uses the age and size parameters and the 'weighting of age criteria' parameter to calculate a priority for each migration candidate file. The timing of file system scans and the start of migration affects which files are migrated. A detailed description of threshold migration explains how candidate files are chosen and migrated.

Start of HSMMONITOR service

The hsmmonitor service checks disk usage. The service compares the current usage with the high threshold and low threshold values for the volume. If the high threshold is reached, migration starts automatically. The HSM for Windows client migrates files from the migration candidates list. If the candidates list contains only a few files, the files from the candidates list are processed immediately. Other files in the file system might be better candidates, but they must first be placed in the candidates list by a file system scan. Age and size criteria, and weighting of age criteria are applied for the current candidates list. If the space usage of the files that are in the candidates list is not sufficient to reach the low threshold, an emergency scan for more candidates begins.

Scanning the file system

The scan for migration candidates runs periodically, based on a configurable time interval. You can also begin an immediate scan with the scan now option.

Scanning the file system with periodic scan

The first periodic scan starts from the root of the volume and scans up to 10000 files. If less than 10000 files are on the volume, the scan stops at the end of the volume. All files that match the age and size criteria are added to the candidates list. If the scan does not reach the end of the volume, a breakpoint (last file scanned) is saved in memory. The breakpoint is saved in memory until the hsmmonitor service stops running.

The next periodic scan starts after the time specified in the migration candidates scanning interval (h) field elapses. If there are less than 10000 files on the volume, the entire volume is scanned.

If a breakpoint exists (the volume contains more than 10000 files), the periodic scan starts from the breakpoint and scans the next 10000 files. If the periodic scan reaches the end of the volume but has not scanned 10000 files, the scan continues from the root.

Until the entire volume has been scanned, the best candidates for migration might not be in the candidates list. When a threshold migration starts, the best candidates from the list are migrated.

Scanning the file system with option 'Scan now'

When you use the scan now option, the hsmmonitor service checks the candidates list before the scan starts. The scan depends on the space usage of the files that are in the candidates list.

If the candidates list contains enough candidates to reach the low threshold, a scan begins immediately. If a breakpoint exists, the scan starts from the breakpoint. If no breakpoint exists, the scan starts from the root. The scan analyzes 10000 files. If there are less than 10000 files on the volume, the scan stops when the entire volume is scanned. If the scan reaches the end of the volume but has not scanned 10000 files, the scan continues from the root. All files that match the age and size criteria are added to the candidates list. If the scan does not reach the end of the volume, the breakpoint is saved in memory. The breakpoint is saved in memory until the hsmmonitor service stops running. The next periodic scan starts after the time specified in the migration candidates scanning interval (h) field elapses.

If the candidates list does not contain enough candidates to reach the low threshold, a demand scan starts. If a breakpoint exists, the demand scan starts from the breakpoint. If no breakpoint exists, the demand scan starts from the root. A demand scan continues until enough files have been added to the candidates list to reach the low threshold during a threshold migration. If the scan reaches the end of the volume but the candidates lists does not have enough files, the scan continues from the root. If the entire volume has been scanned, the scan stops.

The scan now option does not trigger threshold migration.

Scanning the file system during a threshold migration

An emergency scan is triggered if, during a threshold migration, there are not enough candidates in the list to reach the low threshold. If a breakpoint exists, the emergency scan starts from the breakpoint. If no breakpoint exists, the emergency scan starts from the root. An emergency scan continues until enough files have been added to the candidates list to reach the low threshold during a threshold migration. If the emergency scan reaches the end of the volume but the candidates list does not have enough files, the scan continues from the root. If the entire volume has been scanned, the scan stops. If the threshold migration stops, the scan also stops.

Priority of candidates for migration

The files in the candidates list are sorted according to their priority for migration. The priority is determined by comparing a file's age and size with the minimum age and size requirements for migration. A file that is much older or larger than the minimum requirement has a higher priority. A file's age weight is greater when there is a greater difference between a file's age and the minimum age requirement for migration. Similarly, a file's size weight is greater when there is a greater difference between a file's size and the minimum size requirement for migration. For example, a file that is twice as old as the minimum age has an age weight of 2. If the file is exactly the minimum size, it has a size weight of 1.

The priority is also affected by the weighting of age criteria option. A value that is greater than 50 gives the age criteria more weight than the size criteria. A value lower than 50 gives the size criteria more weight than the age criteria.

Priority = (weighting of age criteria /100 * file's age weight) + ((100 - weighting of age criteria)/100) * file's size weight)

High Threshold is reached

When the high threshold is reached, the start of threshold migration is triggered. A bundle of approximately 1000 of the highest priority files are migrated in one run. The exact number of files in a bundle is also determined by the size of the files in the bundle. The bundled files are sorted in alphabetical order for performance optimization. After one bundle is migrated, another bundle is created, sorted, and migrated. The threshold migration process continues until the low threshold is reached or no more candidates are available.

Low Threshold is reached

When the low threshold is reached, the threshold migration stops. All files from the candidates list that were not migrated are kept in the list. If no scan runs before the next threshold migration starts, no files are added to the candidates list. When the next threshold migration runs, the files in the candidates list are prioritized and migrated. It is possible that older and larger files are not in the candidates list.