Accessing an encrypted remote file system using keys from Vault KMIP Secrets Engine

See an example of how to access an encrypted file in a remote cluster.

This topic shows how to configure a cluster so that it can mount an encrypted file system that is in another cluster. In the examples in this topic, the encrypted file system is fs1 and its cluster is Cluster1. The cluster that mounts the encrypted file system is Cluster2.

The examples assume that Cluster1 and fs1 are the cluster and file system that you configured in the topic Setup using HashiCorp Vault KMIP Secrets Engine. You configured Cluster1 for encryption and you created a policy that caused all the files in fs1 be encrypted.

To configure Cluster2 with remote access to an encrypted file in Cluster1, you must configure Cluster2 for encryption in much the same way that Cluster1 was configured. As the following table shows, Cluster2 must add the same key server, create role in the same scope name and register the role using the same RKM stanza name.

Note: In the third column of the table, items in square brackets are connected or added during this topic. The fourth column shows the step in which each item in the third column is added.

Table 1. Setup of Cluster1 and Cluster2
Item	`Cluster1`	`Cluster2`	Steps
File systems	`fs1`	[`fs1_remote`]	Step 1
Connected to key server	`tru-4pub.fyre.ibm.com`	[`tru-4pub.fyre.ibm.com`]	Step 2
Scope name	`spectrumscale` (default)	[`spectrumscale`](default)	Step 3
Created role	`gpfsAdmin`	[`gpfsAdminRemote`]	Step 4
Registered the role to RKM stanza	`gpfsAdmin` to `gpfsRKMstanza`	[`gpfsAdminRemote` to `gpfsRKMstanza`]	Step 4
Has access to master encryption keys	`gpfsAdmin` role in `spectrumscale` scope	[`gpfsAdmin` role in `spectrumscale`]	Step 4
Has access to encrypted file	Local access to testFile file in `fs1` file system	[Remote access to file testFile in file system `fs1`]	Step 5

The encrypted file testFile is in fs1 on Cluster1. To configure Cluster2 to have remote access to file testFile, follow the below steps:

From a node in Cluster2, connect to the remote Cluster1:
1. To set up access to the remote cluster and file system, follow the instructions in topic Accessing a remote GPFS file system.
2. Run the mmremotefs add command to make the remote file system fs1 known to the local cluster, Cluster2:
  
  Note: fs1_Remote is the name by which the remote file system fs1 is known to Cluster2.
```
# mmremotefs add fs1_remote -f fs1 -C Cluster1.gpfs.net -T /fs1_remote -A no
mmremotefs: mmsdrfs propagation completed.
```
  Note: After you have completed Step 1(b) and mounted the remote file system, if you try to access the contents of file testFile from Cluster2, the command fails because the local cluster does not have the master encryption key for the file:
```
# mmmount fs1_remote
Wed Jul 13 00:31:32 EDT 2022: mmmount: Mounting file systems ...

# cat /fs1_remote/testFile
cat: /fs1_remote/testFile: Operation not permitted

mmfs.log:
2022-07-13_00:31:53.456-0400: [E] Unable to open encrypted file: inode 65792, fileset 0, file system fs1.
2022-07-13_00:31:53.456-0400: [E] Key 'leCTiYYS6fUPCgQsk5SHFBtTgADJHgax:gpfsRKMstanza' could not be fetched. The specified RKM ID does not exist; check the RKM.conf settings.
```

From a node in Cluster2, connect to the Vault key server, tru-4pub.fyre.ibm.com, that Cluster1 is connected to.

Run the mmkeyserv server add to connect to tru-4pub.fyre.ibm.com:

# mmkeyserv server add tru-4pub.fyre.ibm.com --auth-token /var/mmfs/ssl/keyServ/tmp/vaultTempToken

Verify that the connection is succeeded:

# mmkeyserv server show
tru-4pub.fyre.ibm.com
        Type:                         KMIP
        IPA:                          9.46.79.137
        User ID:                      N/A
        REST port:                    8200
        Label:                        1_tru-4pub
        NIST:                         on
        FIPS1402:                     off
        Backup Key Servers:           
        Distribute:                   yes
        Retrieval Timeout:            60
        Retrieval Retry:              3
        Retrieval Interval:           10000
        REST Certificate Expiration:  N/A
        KMIP Certificate Expiration:  N/A

From a node in Cluster2, create a role in the same scope, spectrumscale (the default), that Cluster1 created.

Create the gpfsAdminRemote role:

# mmkeyserv role create gpfsAdminRemote --server tru-4pub.fyre.ibm.com --auth-token tempToken
Create a pass phrase for keystore: 
Confirm your pass phrase: 
mmkeyserv: mmsdrfs propagation completed.

Verify that the role is created:

# mmkeyserv role show
gpfsAdminRemote
        Key Server:                 tru-4pub.fyre.ibm.com
        Scope:                      spectrumscale
        Role Label:                 1_gpfsAdminRemote
        RKM Id:                     
        CA Chain Expiration:        2032-05-11 12:40:00 (-0400)
        Certificate Expiration:     2025-07-12 00:59:47 (-0400)
        Certificate Serial Number:  302209357934438088530728828729422923497833539651
        Certificate Type:           system-generated

From a node in Cluster2, register the gpfsAdminRemote role.

The RKM ID must be the same as the one that Cluster1 uses, to allow files created with that RKM ID on Cluster1 to be accessed from Cluster2. However, some of the information in the RKM stanza is different:

# mmkeyserv role register gpfsAdminRemote --rkm-id gpfsRKMstanza
mmkeyserv: mmsdrfs propagation completed.

Verify that the role shows that it is registered:

# mmkeyserv role show
gpfsAdminRemote
        Key Server:                 tru-4pub.fyre.ibm.com
        Scope:                      spectrumscale
        Role Label:                 1_gpfsAdminRemote
        RKM Id:                     gpfsRKMstanza
        CA Chain Expiration:        2032-05-11 12:40:00 (-0400)
        Certificate Expiration:     2025-07-12 00:59:47 (-0400)
        Certificate Serial Number:  302209357934438088530728828729422923497833539651
        Certificate Type:           system-generated

You can display the contents of the new RKM stanza:

# mmkeyserv rkm show
gpfsRKMstanza {
  type = KMIP
  kmipServerUri = tls://9.46.79.137:5696
  keyStore = /var/mmfs/ssl/keyServ/roleCred.1_gpfsAdminRemote.1.p12
  passphrase = pass!@#ForDemo
  clientCertLabel = 1_gpfsAdminRemote
}

You can also view the RKM stanza by displaying the contents of the RKM.conf file on the command-line console:

# cat /var/mmfs/ssl/keyServ/RKM.conf
gpfsRKMstanza {
  type = KMIP
  kmipServerUri = tls://9.46.79.137:5696
  keyStore = /var/mmfs/ssl/keyServ/roleCred.1_gpfsAdminRemote.1.p12
  passphrase = pass!@#ForDemo
  clientCertLabel = 1_gpfsAdminRemote
}

You can now access the encrypted the testFile file remotely from Cluster2:

Verify that you can access the contents of the testFile file:
```
# cat /onFilefs1_remote/testFile
Hello World!
```

Display the encryption attributes of the file:

# mmlsattr -n gpfs.Encryption /onFilefs1_remote/testFile
file name:            /onFilefs1_remote/testFile
gpfs.Encryption:      "EAGC????*Z????????????? ??????C>yA?????????????? ?2+]???ie???aq?q?Am?W0?Q??\?8?@???t??1UN?!?leCTiYYS6fUPCgQsk5SHFBtTgADJHgax?gpfsRKMstanza?"
EncPar 'AES:256:XTS:FEK:HMACSHA512'
        type: wrapped FEK  WrpPar 'AES:KWRAP'  CmbPar 'XORHMACSHA512'
                leCTiYYS6fUPCgQsk5SHFBtTgADJHgax:gpfsRKMstanza

You can now access encrypted files on fs1_remote from Cluster2.