Configuring separate encryption keys for each remote cluster

Starting with IBM Guardium Key Lifecycle Manager 5.0 on IBM Storage Scale 6.0.x, you can configure separate encryption keys for each remote cluster.

Before you begin

Table 1. Preparing storage cluster prerequisites for fileset multitenancy
Prerequisite	Description	Actions to prepare for the configuration of fileset multitenancy
The cluster nodes are installed and the IBM Storage Scale cluster is configured and running.	The mmlscluster command displays the cluster information.	Select a set of contact nodes from the list to share them with the tenant administrator.
	The cluster is healthy.	Use the mmhealth command or the mmgetstate command to corroborate the storage cluster state.
	A cluster public key exists at /var/mmfs/ssl/id_rsa.pub.	Copy the public key file to share it with the tenant administrator.
Encryption server and client are configured.	This storage cluster encryption is an initial setup process that must be done only once.	If the storage cluster does not have an encryption server and a client configured, follow these steps. Use the mmkeyserv command to add the IBM Guardium Key Lifecycle Manager (GKLM) server. When requested, enter the password for the `sklmadmin`. Tip: Configure a backup key server to retain file system access if the primary key server is unavailable. The hostname of the backup key server can be specified by using the --backup parameter. `mmkeyserv server add <GKLM-hostname> [--backup <GKLM-backup-hostname>]` Create a client. Tip: Give a meaningful name to the client, like `storagecluster01`. `mmkeyserv client create <client-name> --server <GKLM-hostname>` Create a tenant. `mmkeyserv tenant add <tenant-name> --server <GKLM-hostname>`

To enable multitenancy for a fileset, the tenant cluster must comply with the prerequisites that are described in the following table.

Table 2. Preparing tenant cluster prerequisites for fileset multitenancy
Prerequisite	Description	Actions to prepare for the configuration of fileset multitenancy
The cluster nodes are installed and the IBM Storage Scale cluster is configured and running.	The mmlscluster command displays the cluster information.	None.
	The cluster is healthy.	Use the mmhealth command or the mmgetstate command to corroborate the tenant cluster status.
	A cluster public key exists at /var/mmfs/ssl/id_rsa.pub.	Copy the public key file to share it with the storage administrator.
Network connectivity between the tenant cluster and the storage cluster.	The daemon IP addresses and hostnames of the storage cluster can be resolved and reached from the tenant cluster.	For example, the next command pings three times: `ping -c 3 -w 1 <node_name>`.
	The daemon IP addresses of the tenant cluster are reachable from all the nodes in the storage cluster.	For example, the next command pings three times: `ping -c 3 -w 1 <node_name>`.
Restricted roles for IBM Guardium Key Lifecycle Manager user.	These are used in substep 1a.	The required restricted roles for a GKLM user are `klmClientUser`, `klmConfigure`, and `klmAdminDeviceGroup`.

Procedure

On the tenant cluster, request tenant registration.

These substeps are to request registration of a tenant cluster in the storage cluster and to allow the tenant cluster to establish a remote cluster relationship. These substeps include the request to register a GKLM key client, which is needed to set up encryption by using tenant-specific encryption keys.
1. Use the mmkeyserv command to add the GKLM server by using the GKLM user and password that was provided by the storage administrator.
  Tip: Configure a backup key server to retain file system access if the primary key server is unavailable. The hostname of the backup key server can be specified by using the --backup parameter.
```
mmkeyserv server add <GKLM-hostname> --user-id <GKLM-user> [--backup <GKLM-backup-hostname>]
```
2. Add the tenant by using the tenant name as it was provided to the storage administrator.
```
mmkeyserv tenant add <tenant-name> --server <GKLM-hostname>
```
3. Create a client.
  As a best practice, append the word client to the client name (for example, tenant91client).
```
mmkeyserv client create <client-name> --server <GKLM-hostname>
```
4. Submit a request to GKLM to register the client to the tenant.
```
mmkeyserv client register <client-name> --tenant <tenant-name> --rkm-id <remote-key-management-ID> --noaccept
```
  Example output for mmkeyserv client register
  Waiting for acceptance The security administrator of gklm-server42.com must accept the pending request through the SKLM GUI. Please use the following Certificate Name when prompted by the GUI: 7629575310756386022_tenant91_1757501789 Once the pending request is accepted, restart the command.
5. Share the printed cert name with the storage administrator.
  
  An example cert name is: 7629575310756386022_tenant91_1757501789. The request must be accepted by the storage administrator (or the GKLM administrator) by using the GKLM GUI or native REST API.
On the storage cluster, accept the tenant registration.

These substeps are to configure encryption for a new tenant on the storage cluster, and to authorize the remote cluster access.
1. Use the mmkeyserv command to add the tenant by using the tenant name as it was provided by the tenant administrator.
  When requested, enter the password for the sklmadmin.
```
mmkeyserv tenant add <tenant-name> --server <GKLM-hostname>
```
2. Register the tenant to the client on the storage cluster.
  Use the mmkeyserv command to show existing clients.
```
mmkeyserv client register <client-name> --tenant <tenant-name> --rkm-id <remote-key-management-ID>
```
3. Create an encryption key for the tenant.
  Remember: Save the key UUID output, you need it in a later step (5.d) in this procedure.
```
mmkeyserv key create --server <GKLM-hostname> --tenant <tenant-name>
```
4. Establish and authorize a remote cluster relationship between the tenant cluster and the storage cluster.
  The storage administrator needs to authorize the tenant cluster to access the storage cluster. To check the tenant cluster name, consult the public key file.
```
mmauth add <tenant-cluster-name> -k <id.pub-file-from-tenant>
```
On the tenant cluster, complete the request for tenant registration.

These substeps are to add the remote cluster relationship and mount on the tenant cluster. After the storage administrator authorizes the tenant cluster, the tenant administrator must add the remote cluster relationship by using the public key file, and contact the nodes that were provided by the storage administrator. To check the storage cluster name, consult the public key file.
1. Make sure that all the contact node hostnames can be resolved from all the tenant cluster nodes.
```
mmremotecluster add <storage-cluster-name> -n <contact-node> -k <id.pub-file-from-storage-cluster>
```
2. Wait for the request to be accepted by the storage cluster administrator.
  
  This is the request made in substep 1e.
3. To complete the registration, run again the command to register the GKLM key client.
```
mmkeyserv client register <client-name> --tenant <tenant-name> --rkm-id <tenant-name> --noaccept
```
Request to provision storage from the storage cluster to the tenant cluster.

The administrator of the tenant cluster must contact the administrator of the storage cluster to make this request. The goal is to communicate the required capacity, bandwidth, and related resources to the storage administrator.
On the storage cluster, accept the request for tenant storage.
1. Provision a new tenant fileset in the file system of the storage cluster.
  The storage administrator creates and links the tenant fileset by running the following commands.
```
mmcrfileset <file-system-name> <tenant-fileset-name> --inode‐space new
mmlinkfileset <file-system-name> <tenant-fileset-name> -J /<file-system-name> /<tenant-fileset-name>
```
2. Set quotas to limit tenant capacity and inode usage.
  Configure the quota limits according to the requested capacity (per block) and inode limit (for files) that were specified by the tenant administrator.
```
mmsetquota <file-system-name>:<tenant-name> --block block_Q_soft:block_Q_hard --files file_Q_soft:file_Q_hard
```
3. According to the limits requested by the tenant administrator, configure QoS bandwidth to avoid monopolization of shared storage input/output resources.
  Create a QoS class for the tenant fileset and create a throttle with the requested limits.
```
mmqos class create <file-system-name> --class <tenant-name> --fileset <tenant-name>
mmqos throttle create <file-system-name> --pool all --class <tenant-name> --maxMBS <requested-bandwidth>
```
4. Set the policy for fileset level encryption on the file system.
  Create a fileset encryption rule by using the provided tenant name and the key UUID that was created in a previous step (2.c).
  Example for creating a fileset encryption rule
  RULE 'Rule-<tenant-name>' ENCRYPTION 'E_<tenant-name>' IS ALGO 'DEFAULTNISTSP800131A' KEYS('<key-UUID>:<tenant-name>') RULE 'Encrypt fileset <tenant-name> with rule E_<tenant-name>' SET ENCRYPTION 'E_<tenant-name>' FOR FILESET ('<tenant-name>')
5. Install the policy for fileset level encryption into the file system.
  1. Retrieve the existing policy by issuing the mmlspolicy command into a policy file.
```
mmlspolicy <file-system-name> -L > polfile.txt
```
  2. Use a text editor to append the new rule to the policy file.
  3. Use the mmchconfig command to install the new policy.
```
mmchpolicy <file-system-name> <policy-file>
```
6. Use remote fileset access control (RFAC) to authorize remote file system access, and thus allow the tenant cluster to access the new tenant fileset.
  The storage administrator must grant access to the tenant cluster to access the fileset that was created for the tenant.
```
mmauth grant <tenant-cluster-name> -f <file-system-name> --fileset root,<tenant-name>
```
On the tenant cluster, complete the storage request.

These substeps are to add the remote file system and mount it.
1. The tenant administrator must add the remote file system by using a file system name of choice (in the next example, tenantfs).
```
mmremotefs add /dev/tenantfs -f <file-system-name-from-storage-administrator> -C <storage-cluster-name>
```
2. Mount the file system so that it can be accessed on the tenant cluster.
```
mmmount /dev/tenantfs cd /mnt/tenantfs/<tenant-name>
```