IBM Storage Scale with thin provisioned devices

IBM Storage Scale provides features to reduce the risks of thin provisioning and to optimize storage use in thin provisioned storage systems.

With thin provisioning, storage space can be optimized by allocating space on demand and reclaiming when the application that is using a given volume declares that it is no longer using a given block of data. The storage system allocates storage to applications, including file systems, only when the storage is needed, from a pool of free storage. Thin provisioning addresses the problem of under-utilization of available storage system capacity. System administrators do not have to monitor and manage how much storage is required by each application. Instead, thin provisioning lets administrators provision large "thin" LUNs to a host based on estimated and anticipated usage without reserving physical space in advance. When the application writes data, the storage system allocates physical storage from the free pool on the storage system to the thin-provisioned LUNs.

Thin provisioning of storage for file systems can provide an excellent alternative in increasingly cost-conscious data centers by allowing storage hardware purchases to be delayed and capacity to be purchased just in time.

A significant concern with data reduction storage is the possibility that the device might run out of physical space on a thin provisioned volume. For a file system running on such a device, the expectation is that the volume has been fully allocated, but if the storage has been over-provisioned, it is possible that no physical space exists to accommodate a write request. The result is that the device returns an I/O error, an event that can cause the file system to unmount and become inaccessible. In the case of an IBM Storage Scale file system, remounting the file system requires running through log recovery, a process that itself requires writing into the file system, and that might again fail if the device has no physical disk space available.

To minimize the possibility of the device running out of physical space, the device needs to be vigilantly monitored; otherwise, the benefits of thin provisioning might be offset by the possibility that the device runs out of space and brings the file system offline.

Attention: Data reduction in IBM Storage Scale, including thin provisioned volumes, is supported only after a technical compatibility review by IBM®. Ask your sales representative to contact IBM Storage Scale development about the RPQ or SCORE process.

Enhancement in IBM Storage Scale

IBM Storage Scale provides several features to mitigate the risks of supporting thin provisioning and to optimize storage use on thin provisioned storage systems:

Space reclamation: This operation ensures that disk blocks that are not used by the file system are returned to the storage device.
Emergency disk space management: Space is reserved in the file system, in the form of metadata files with disk blocks filled with incompressible data, which can be freed back to the storage on demand.

Table 1. Thin provisioning storage solutions in IBM Storage Scale
Feature	Description	Benefits
Automatic recovery space. For more information, see the description of the three types of recovery space in the subtopic Deployment recommendation.	Automatic recovery space is reserved by the file system, which enables IBM Storage Scale to recover the storage system if it runs low on physical space.	This feature reduces system administrator workload to estimate how much additional physical space is needed to prevent a file system from becoming inaccessible when a storage system runs low on physical space.
The recovery log flush operation is thin-space aware.	The recovery log flush operation ensures that enough reserved space is always available to run log recovery.	This feature ensures that a file system can be recovered after a storage system runs out of physical space.
Thin space reclamation is supported.	The mmreclaimspace command reclaims unused space in a file system and returns it to the storage system.	This feature improves storage space utilization and savings.
Thin space emergency recovery.	Free the previously reserved space to allow the storage system to acquire enough space for file system recovery.	This feature recovers file systems after a failure caused by an out-of-physical-space condition in the storage system.

Deployment recommendation

Three types of emergency recovery space are used:

Required recovery space: You must reserve this space before you create an IBM Storage Scale file system. Create several volumes in the storage system and fill them with uncompressible data. In case the storage system runs out of physical space, the required recovery space enables IBM Storage Scale to recover the volumes in the storage system back to a read/write state. For more information, see the subtopic During system configuration later in this help topic.
Optional recovery space: It is a good practice to also create several additional recovery volumes and fill them with uncompressible data. If the storage system runs nearly out of physical space, the optional recovery space enables IBM Storage Scale to prevent the storage system from running completely out of space. For more information, see the subtopic During system configuration later in this help topic.
Automatic recovery space: IBM Storage Scale reserves this space for a file system the first time that the file system is mounted. This type of space enables IBM Storage Scale to recover the file system if the storage system runs out of space. However, this type of space can be released to IBM Storage Scale only while the device is operational and in read/write mode. It cannot be released if the file system has become either read-only or offline because the storage system has run completely out of space. For more information, see Emergency recovery.

The following subtopics describe the steps that you should take for thin provisioning support during system configuration, during ongoing operations (production), and during emergency recovery after the storage system has run nearly or completely out of physical space.

During system configuration

During system configuration you must reserve a required recovery space and you can also create an optional recovery space. For the meaning of these terms, see the previous subtopic.

To create the required recovery space, create several emergency recovery volumes in the storage system and fill them with uncompressible data, such as fully random data that cannot be subject to deduplication. The size of this space varies depending on the type and brand of the storage system. For more information about the amount of reserved space that is needed, contact the vendor of the storage system. If you are using an IBM FlashSystem A9000, A9000R or any IBM Storage Virtualize systems that use IBM FlashCore Modules (FCMs), and the system runs out of space, contact IBM Storage Virtualize support team for assistance.

If the storage system runs out of physical space, IBM Storage Scale uses this required recovery space to bring the storage system back online. For more information, see Emergency recovery.

After you create the required recovery space, you must create LUNs in the storage system, export them, and specify each LUN in an nsd stanza in the IBM Storage Scale stanza file. See the following example:

%nsd:
    nsd=gpfs1nsd
    usage=metadataOnly
    pool=system
    thinDiskType=scsi

The line thinDiskType=scsi is required and indicates that the device is a thin provisioned disk.

Note: The device must support the SCSI WRITE SAME command.

For more information about the thinDiskType parameter, see the topics mmcrnsd command and mmcrfs command.

To create the optional recovery space, create several volumes in the storage system and fill them with uncompressible data. A suggested size for the emergency recovery space is 20 GiB times the sum of all the local and remote nodes that can mount each of the file systems that is on the storage system. In other words:

20 GiB x ((the number of local and remote nodes that can mount FS1) + 
          (the number of local nodes that can mount FS2) + 
          .... + 
          (the number of local and remote nodes that can mount FSN))

If the storage system runs nearly out of physical space, IBM Storage Scale can use this optional recovery space to prevent the storage system from running completely out of physical space. For more information, see the subtopic Emergency recovery later in this help topic.

During production

During production you must carefully monitor the amount of physical space that is available in the storage system. The vendor of the storage system can probably provide utility programs for monitoring available physical space. If you determine that the storage system is running nearly out of physical space, follow the next steps. (If the storage system runs completely out of physical space, follow the instructions in the Emergency recovery subtopic later in this help topic.)

Quiesce all applications that are writing data into the storage system.
Delete unused volumes from the storage system.
Delete unused files and directories in the IBM Storage Scale file systems.
If you created optional recovery space during system configuration, free it. With this freed space IBM Storage Scale can recover the storage system before it runs completely out of space.
Issue the mmreclaimspace command to reclaim physical space in the storage system that is not in use but is not marked as available:
```
mmreclaimspace Device --reclaim-threshold 0
```
For more information, see mmreclaimspace command.
Note: Because the space-reclaiming operation consumes a great deal of CPU, memory, network, and I/O system resources, it is a good idea to run the mmreclaimspace command at a time when the system is not heavily loaded.
Expand the free physical space in the storage system by adding new disks or deleting existing volumes.

If the storage system now has enough available physical space, you can resume all applications. If you freed the optional recovery space in Step 4, it is a good practice to re-create it as soon as possible.

Emergency recovery

If the storage system runs completely out of physical space, the volumes that use the storage system are likely to go offline or become read-only. To recover the system storage and to recover the IBM Storage Scale file systems, follow the procedure that is outlined next. This procedure has three main parts:

Part 1: Bring the file system to a state in which data can be read although data is not written. Follow these steps:

Stop all applications that are writing into the storage system.
Disable the operating system automount feature by issuing the appropriate system command:
- On Linux®, issue the following command:
```
systemctl disable autofs
```
- In AIX®, issue the following command:
```
stopsrc -s automountd 
```
Issue the following command to unmount all IBM Storage Scale file systems:
```
/usr/lpp/mmfs/bin/mmumount all -a
```
Release the required recovery space that you reserved during system configuration. All volumes should be operational and in read/write mode.
Issue the following command to correct the states of the NSD disks. This action allows the mount operation in Step 7 to succeed across the cluster:
```
/usr/lpp/mmfs/bin/mmnsddiscover -a -N all
```
If you created optional recovery space during system configuration, free it.
Remount the file system in read-only mode. If the mount succeeds, skip the next step and go on to Part 2. IBM Storage Scale reserves the automatic recovery space as the file system is mounted for the first time.
If the mount operation in the preceding step fails, follow these steps to mount the file system:
1. Issue the following command to mount the file system in restricted mode:
```
/usr/lpp/mmfs/bin/mmmount Device -o rs
```
2. Issue the following command to release the automatic recovery space that IBM Storage Scale reserved when it created the file system:
```
/usr/lpp/mmfs/bin/mmreclaimspace Device --emergency-reclaim
```
  Note: The --emergency-reclaim option is valid only if the thin-provisioned storage is read-writable and the file system is mounted in restrict mode. For more information, see mmreclaimspace command.
3. Remount the file system in read-only mode as in Step 7. The file system should be mounted successfully.

Part 2: Expand the free physical space of the storage system. You can add disks, delete data, or move data to other file systems.

Part 3: Bring the file system to a state in which data can be both read and written. Follow these steps:

Reserve required recovery space in the physical storage system as you did during system configuration.
If you freed optional recovery space in Step 6 of Part 1, re-create it as you did during system configuration.
Issue the following command to mount the file system in read/write mode:
```
/usr/lpp/mmfs/bin/mmmount Device
```
If you ran the mmreclaimspace command in Step 8(b) of Part 1, IBM Storage Scale reserves automatic recovery space during this mount operation.
Enable the operating system automount feature by issuing the appropriate system command:
- On Linux, issue the following command:
```
systemctl enable autofs
```
- In AIX, issue the following command:
```
startsrc -s automountd
```

If Step 3 is completed without any error, the file system is writable again. If an error occurred, the free physical space that you expanded in Part 2 is not enough. Repeat the emergency recovery process, beginning with Part 1, Step 1.

Graceful emergency recovery

Starting with release 5.2.0, IBM Storage Scale offers this additional emergency recovery feature for thin-provisioned storage systems, LUNs, and volumes created from FCM drives.

The graceful emergency recovery procedure provides a mounting mode for space reclaim (mount option -o sr), which allows system administrators and users to delete files to expand the free physical space for the system.

To recover the storage system and the IBM Storage Scale file system by using the graceful emergency recovery, follow the procedure outlined next. This procedure has three main parts.

Part 1: Bring the file system to a state in which it can be mounted in restricted mode: then, in preparation for the next step (where files can be accessed for deletion and removal), release the automatic recovery space. Follow these steps:

Stop all applications that are writing into the file system.
Issue the following command and make sure that the file system is unmounted (if it is not already unmounted by the SGPanic() procedure).
```
# /usr/lpp/mmfs/bin/mmumount Device -a
```
Mount the file system in restricted mode. Then, release the automatic recovery space that gets reserved when the file system is mounted in regular I/O mode.
```
# /usr/lpp/mmfs/bin/mmmount Device -o rs
```
To release the automatic recovery space:
```
# /usr/lpp/mmfs/bin/mmreclaimspace Device --emergency-reclaim
```
Unmount the file system.
```
# /usr/lpp/mmfs/bin/mmumount Device
```

Part 2: Mount the file system with the space reclaim mode, -o sr. Then, still in this mode, users must delete files and expand the free physical space of the storage system. Follow these steps:

Mount the file system in space reclaim mode.
```
# /usr/lpp/mmfs/bin/mmmount Device -o sr
```
Expand the free physical space of the file system by deleting (or moving out) the files in the file system.
Note: If a snapshot is created, make sure to also delete all of them (for more information, see the mmdelsnapshot command). Otherwise, the free physical space is not recognized properly due to the snapshot effects.
Reclaim free spaces so that all of them are recognized by the storage system.
Note: The mmreclaimspace command would fail while the background space reclaim is running.

Note: If background space reclaim is enabled on the file systems, make sure to wait until the operation is completed. Check mmfs.log.latest, where you must see a message that displays an output similar to the following one:
```
# grep background /var/adm/ras/mmfs.log.latest
2024-02-02_13:06:37.543-0500: [I] Initializing background space reclaimation for 'gpfs'...         
2024-02-02_13:10:55.015-0500: [I] Exit background space reclaimation for 'gpfs'...
```
After enough physical space is free, unmount the file system.
Note: The size of the free physical space can be subjective. It is important to make enough free space (or add more disks to the storage system) for the file system to prevent encountering the OOS (Out-Of-Space) condition soon.
```
# /usr/lpp/mmfs/bin/mmumount Device
```

Part 3: Bring the file system to a state in which data can be both read and written. Follow these steps:

Issue the following command to mount the file system in regular read/write mode.
```
# /usr/lpp/mmfs/bin/mmmount Device -a
```
If preferred, enable the operating system automount feature by issuing the appropriate system command.
```
# systemctl enable autofs
```

If the graceful emergency recovery procedure is completed without any error, the file system is writable again. If an error occurs or if the free physical space that you expanded in Part 2 is not enough, repeat the graceful recovery procedure, beginning with Part 1, Step 1.