Protecting file data: IBM Spectrum Scale safeguarded copy

IBM Spectrum Scale 5.1.5 introduces the safeguarded copy (SGC) as a mechanism to protect file system data.

The safeguarded copy (SGC) is a mechanism to protect data in IBM Spectrum Scale file systems. It is based on the immutable snapshot feature. The mechanism secures data from deliberate or accidental compromise.

The following facts help to explicitly define the Safeguarded copies:
  • Safeguarded copies (SGCs) are immutable copies of a file system or fileset, and they help minimize the impact of cyberattacks, disasters and failures.
  • SGCs are snapshots with a retention time, and which cannot be altered, and also cannot be deleted until the retention time is expired.
  • The administrator can use the GUI to schedule periodic snapshot creation, and also snapshot deletion after the retention time has expired.
  • An SGC cannot be modified or deleted by a single bad actor, and the SGC’s retention time cannot be changed by a single bad actor.

Safeguarded copies are used to take frequent snapshots of a production file system. You can take, for example, hourly snapshots that are maintained for a number of days. These snapshots act as backup copies for recovering data if the primary data is corrupted or destroyed. If the content of files is damaged, you can run the mmrestorefs command to restore the production file system or its independent filesets from these immutable snapshots. These immutable snapshots share the storage space with the production file system, and can be accessed online along with other regular snapshots. The system administrator can also make safeguarded copies offline by copying data out to a separate storage system from these immutable snapshots, then restore the production file system from there, on demand.

As a safeguarded copy, an immutable snapshot cannot be modified or deleted. You can set the retention period for a snapshot when the snapshot is created, which prevents the snapshot from being deleted until the retention time is completed or expires. This feature helps protect the data even if the administrator's account is compromised. By periodically creating snapshots with a retention period, one always ensures ready availability of a snapshot to restore the content of the data, if needed.

For security reasons, the safeguarded copy environment must be managed by a non-root administrator to prevent a malicious attacker from acquiring root user privilege by taking over the administrator account, and corrupting the production file system or disks with operations allowed only for root user. Several IBM Spectrum Scale operations require root privilege. Therefore, for a safeguarded copy environment, you must configure the IBM Spectrum Scale sudo wrapper. For more information, see https://community.ibm.com/community/user/storage/blogs/nils-haustein1/2020/12/17/spectrum-scale-sudo-wrappers.

Configuring an IBM Spectrum Scale administrator to run using a non-root account has the benefit of not allowing the administrator to alter the system clock. If allowed to manipulate the system clock, administrators can move the time of day forward past the snapshot's expiration time, and delete the snapshot, therefore defeating the use of snapshot retention time.

In certain cases, such as when the file system runs out of space, it might become necessary to delete a safeguarded copy snapshot even before its retention time expires. In those cases, the mmrestrictedctl command can be used to delete snapshots whose retention time has not expired.

To prevent the IBM Spectrum Scale administrator from removing snapshots until their retention time has expired, the mmrestrictedctl command must be run only with explicit authorization from the security administrator. The sudo rules must also be accordingly constructed. The sudo configuration that is mentioned in the blog https://community.ibm.com/community/user/storage/blogs/nils-haustein1/2020/12/17/spectrum-scale-sudo-wrappers instructs the mmrestrictedctl command to remain disabled for normal administrators. It becomes available only after the security administrator temporarily adds the command to the allowed list of commands for the IBM Spectrum Scale administrator. For more information, see mmrestrictedctl command .

Data protection using safeguarded copies

Data protection can be ensured by using the following three workflows:
Establish a safeguarded environment.
Follow the steps to establish a safeguarded copy environment:
  1. Prepare the sudo administration environment. For more information, see https://community.ibm.com/community/user/storage/blogs/nils-haustein1/2020/12/17/spectrum-scale-sudo-wrappers.
  2. In the IBM Spectrum Scale GUI, schedule periodic snapshot creation with retention time, and deletion of those snapshots after the expiration time has elapsed. For more information, see Creating immutable snapshots using the GUI.
Use mmrestorefs command to restore the content of damaged files
Follow the steps to restore damaged files
  1. Stop the applications that are using the corresponding files, filesets or file systems.
  2. Use the mmrestorefs command to restore data from a snapshot that contains good data into the file system or fileset.
Delete snapshots before their expiration time owing to an emergency
Follow the steps to delete snapshots before their expiration time, in case it becomes necessary.
  1. The IBM Spectrum Scale administrator contacts the security administrator and requests to be granted permission to run the mmrestrictedctl command.
  2. When permission is granted, the IBM Spectrum Scale administrator runs the mmrestrictedctl command to delete snapshots which have still not expired. The number of snapshots being deleted must be only as few as required.
  3. The IBM Spectrum Scale Scale administrator contacts the security administrator to remove the permission to run the mmrestrictedctl command.
Note: The operations that are enabled by the mmrestrictedctl command can only be used by running this command and must be used only for certain unusual cases like the ones mentioned. The IBM Spectrum Scale management GUI does not provide any features to run this command.

Creating immutable snapshots using the GUI

To streamline the management of safeguarded copies, you can configure a rule to automatically create and delete immutable snapshots with the desired frequency and retention period through the IBM Spectrum Scale management GUI.

To schedule snapshot creation and retention, perform the following steps:

  1. Log in to the IBM Spectrum Scale GUI and select Files > Snapshots.
  2. Click Create Snapshot.
  3. In the Create Snapshot window, type the path of the file system or independent fileset for which you need to create snapshots.
  4. In the Snapshot name field, type the name of the snapshot.
  5. Click Create Rule to schedule the snapshot creation and retention. The Create Snapshot Rule window is displayed.
  6. In the Name field, type the name of the snapshot scheduling rule.
  7. In the Frequency field, select the frequency in which you need to create snapshot. You need to enter some more details based on the value that is selected in the field. For example, if the Multiple Times an Hour value is selected, then select the minutes of the hour in which you need to create snapshots.
  8. In the Retention fields, select the number of snapshots that must be retained in a period.
  9. In the Deletion Schedule field, select whether you want the snapshots to be deleted immediately after creation or during off-hours
  10. In the Prefix field, specify a prefix to be added with the name of the snapshots that are created with this rule. The prefix is added to the date and time to identify the rule that is used to create the snapshot. If a prefix is not specified, the default prefix @GMT is used. Using the default prefix enables the Microsoft Windows Volume Shadow Copy Service (VSS) identification if the file is shared using the SMB protocol.
  11. Select the Allow Expiration checkbox to delete the snapshot when the defined retention period is completed.
    Note: If you do not select the Allow Expiration checkbox then the expiration time for all snapshots created is the same as its creation time.
  12. Click OK to save the changes
  13. In the Create Snapshot window, click Create to create the snapshot creation schedule and retention policy for the snapshots.

Restoring data with mmrestorefs command

If fileset or file system data is damaged or corrupted, it can be restored from one of the snapshots with the mmrestorefs command. If only a few files have been affected then these can be restored by copying them directly from the snapshot to the corresponding file in the file system. The snapshot appears as part of the file system or fileset namespace. For more information, see mmrestorefs command .