Setup using HashiCorp Vault KMIP Secrets Engine

This topic describes how to setup encryption using HashiCorp Vault KMIP Secrets Engine as the RKM key server.

Requirements:
The following requirements must be met on every IBM Storage Scale node that participates in encryption:
  • The node must have direct network access to the system where the key server is installed.
  • The security-sensitive files that are created during the configuration process must have the following characteristics:The security-sensitive files include the following files:
    • The RKM.conf file. For more information about this file, see The RKM.conf file and the RKM stanza.
    • The files in the client keystore directory, which include the keystore file, the public and private key files for the client, and possibly other files. For more information about these files, see The client keystore directory and its files.
      Note: In the simplified setup, the mmkeyserv command automatically creates and distributes the RKM.conf files and the files in the client keystore directory to every node in the cluster. The files are located in the following directory on each node:
       /var/mmfs/ssl/keyServ
  • Client keystore files must be record-locked when the GPFS daemon starts. If the keystore files are stored on an NFS mount, the encryption initialization process can hang. The cause is a bug that affects the way NFS handles record locking. If you encounter this problem, upgrade your version of NFS or store your keystore file on a local file system. If an upgrade is not possible and no local file system is available, use a RAM drive to store the keystore files.
See the following subtopics for instructions:

Part 1: Installing and configuring Vault KMIP Secrets Engine

Follow the below links to install and configure HashiCorp Vault KMIP Secrets Engine:

Part 2: Configuring IBM Storage Scale cluster to use HashiCorp Vault KMIP Secrets Engine for encryption

Gather the following information:
  • The Vault Enterprise key server hostname.
  • Create and save a temporarily authentication token in a file with owner only read permission.
The following table provides a high-level overview of the configuration process. The steps in the table correspond to the steps in the procedure that begins immediately after the table.
Table 1. Configuring the cluster to use HashiCorp Vault KMIP Secrets Engine
Step Actions
1 Verify the direct network connection between the IBM Storage Scale node and the RKM server.
2 Add the RKM key server to the configuration.
3 Create a new role from a scope on the RKM server.
4 Register the role created in the previous step.
5 Create a master encryption key for the role.
6 Set up an encryption policy in the node.
7 Test the encryption policy.
  1. Verify that the IBM Storage Scale node that you are working from has a direct network connection to the RKM server.
  2. Add the RKM server to the encryption configuration:
    1. Use the mmkeyserv server add command to add the RKM server.
      For example:
       mmkeyserv server add tru-4pub.fyre.ibm.com --auth-token tempToken
mmkeyserv: mmsdrfs propagation completed.
      In this example:
      • tru-4pub.fyre.ibm.com is the host name of the Vault Enterprise key server.
      • tempToken contains a temporarily token that given by the Vault administrator.

      If Vault Enterprise is not configured to use the default REST port of 8200, you must specify the –port option and provide the correct port number when adding the server.

      HashiCorp Vault Enterprise supports high availability clusters to protect against outages. High availability mode is automatically enabled when using a data store that supports it.

      HashiCorp recommends Vault Integrated Storage as the default high availability backend for new deployments of Vault. IBM Storage Scale encryption high availability with HashiCorp Vault Enterprise was tested with Vault Integrated Storage backend.

      Configure HashiCorp Vault Enterprise high availability cluster. For more information, see High Availability Mode (HA). Use the --backup parameter and specify a comma-separated list of the standby server names. If an IBM Storage Scale node cannot retrieve a master encryption key from the primary key server, it tries the next server in the list until it either retrieves the key or exhausts the server list.

    2. Issue the mmkeyserv server show command to verify that the key server is added. The following listing shows that tru-4pub.fyre.ibm.com is added: 
      # mmkeyserv server show
tru-4pub.fyre.ibm.com
        Type:                         KMIP
        IPA:                          9.46.79.137
        User ID:                      N/A
        REST port:                    8200
        Label:                        1_tru-4pub
        NIST:                         on
        FIPS1402:                     off
        Backup Key Servers:           
        Distribute:                   yes
        Retrieval Timeout:            60
        Retrieval Retry:              3
        Retrieval Interval:           10000
        REST Certificate Expiration:  N/A
        KMIP Certificate Expiration:  N/A
  3. Issue the mmkeyserv role create command to create a new role from a scope on the RKM server. The command creates the scope on the RKM server if it does not exist.
    • A role name must be unique within an IBM Storage Scale cluster.
    • A role name is case-insensitive and must be 1 to 16 alphanumeric characters.
    • A scope name is case-insensitive and must be 1 to 16 alphanumeric characters.
    • Only one role from an IBM Storage Scale cluster can be created per scope. This makes the scope unique within an IBM Storage Scale cluster.
    For example:
     mmkeyserv role create gpfsAdmin --server tru-4pub.fyre.ibm.com --auth-token tempToken
Create a pass phrase for keystore: 
Confirm your pass phrase: 
mmkeyserv: mmsdrfs propagation completed.

    The above command issued without the --scope option. By default, it created the gpfsAdmin role in the spectrumscale scope.

    Issue the mmkeyserv role show command to verify that the role is created. The following command listing shows the gpfsAdmin role is created under the spectrumscale scope:
     mmkeyserv role show
gpfsAdmin
        Key Server:                 tru-4pub.fyre.ibm.com
        Scope:                      spectrumscale
        Role Label:                 1_gpfsAdmin
        RKM Id:                     
        CA Chain Expiration:        2032-05-11 12:40:00 (-0400)
        Certificate Expiration:     2025-06-30 01:48:44 (-0400)
        Certificate Serial Number:  208366370350887968995911658331713853666704552539
        Certificate Type:           system-generated
  4. Use the mmkeyserv role register command to register the role created in Step 3. You must provide a remote key management (RKM) ID as an input for this command. The RKM ID becomes the identifier field of a new RKM stanza that describes the connection between this client and the key server. For more information about the RKM stanza, see The RKM.conf file and the RKM stanza. This command creates the RKM stanza in RKM.conf file.
    For example:
     mmkeyserv role register gpfsAdmin --rkm-id gpfsRKMstanza
mmkeyserv: mmsdrfs propagation completed.

    Issue the command mmkeyserv role show verify that the role gpfsAdmin is associated with the gpfsRKMstanza RKM ID, as shown in the following example:

     mmkeyserv role show
gpfsAdmin
        Key Server:                 tru-4pub.fyre.ibm.com
        Scope:                      spectrumscale
        Role Label:                 1_gpfsAdmin
        RKM Id:                     gpfsRKMstanza
        CA Chain Expiration:        2032-05-11 12:40:00 (-0400)
        Certificate Expiration:     2025-06-30 01:48:44 (-0400)
        Certificate Serial Number:  208366370350887968995911658331713853666704552539
        Certificate Type:           system-generated
    You can also issue the command mmkeyserv rkm show to verify the RKM stanza created by the mmkeyserv role register command.
     mmkeyserv rkm show
gpfsRKMstanza {
  type = KMIP
  kmipServerUri = tls://9.46.79.137:5696
  keyStore = /var/mmfs/ssl/keyServ/roleCred.1_gpfsAdmin.1.p12
  passphrase = pass!@#ForDemo
  clientCertLabel = 1_gpfsAdmin
}
    The same information should be available in the RKM.conffile on the node:
    # cat /var/mmfs/ssl/keyServ/RKM.conf
gpfsRKMstanza {
  type = KMIP
  kmipServerUri = tls://9.46.79.137:5696
  keyStore = /var/mmfs/ssl/keyServ/roleCred.1_gpfsAdmin.1.p12
  passphrase = pass!@#ForDemo
  clientCertLabel = 1_gpfsAdmin
}
  5. Issue the mmkeyserv key create command to create a master encryption key for the role. The following command displays the UUID of the encryption key (not the key value itself): 
     mmkeyserv key create --role gpfsAdmin
leCTiYYS6fUPCgQsk5SHFBtTgADJHgax
  6. Set up an encryption policy on the node.
    1. Create a file that contains the policy rules that instructs GPFS to do the encryption tasks that you want. The following example policy instructs IBM Storage Scale to encrypt all files in the file system with a file encryption key (FEK) and to wrap the FEK with a master encryption key (MEK): 
      RULE 'p1' SET POOL 'system' /* one placement rule is required at all times */
RULE 'Encrypt all files in file system with rule E1'
   SET ENCRYPTION 'E1'
   WHERE NAME LIKE '%'
RULE 'simpleEncRule' ENCRYPTION 'E1' IS
   ALGO 'DEFAULTNISTSP800131A'
   KEYS('leCTiYYS6fUPCgQsk5SHFBtTgADJHgax:gpfsRKMstanza')
      In the last line of the policy, the character string within single quotation marks (') is the key name. A key name is a compound of two parts in the following format: 
      KeyID:RkmID
      where:
      KeyID
      Specifies the UUID of the key that you created in Step 5.
      RkmID
      Specifies the RKM ID that you specified in Step 4.
    2. Issue the mmchpolicy command to install the rule.
      Note: Installing a new policy with the mmchpolicy command removes all the statements in the previous policy. To add statements to an existing policy without deleting the previous contents, collect all policy statements for the file system into one file. Add the new statements to the file and install the contents of the file with the mmchpolicy command.
      1. Issue the following command to install the policy rules the /tmp/fs1.pol file for the fs1 file system:
         mmchpolicy fs1 /tmp/fs1.pol
Validated policy 'fs1.pol': Parsed 3 policy rules.
Policy `fs1.pol' installed and broadcast to all nodes.
      2. You can list the new encryption policy with the following command:
         mmlspolicy fs1 -L
RULE 'p1' SET POOL 'system' /* one placement rule is required at all times */
RULE 'Encrypt all files in file system with rule E1'
   SET ENCRYPTION 'E1'
   WHERE NAME LIKE '%'
RULE 'simpleEncRule' ENCRYPTION 'E1' IS
   ALGO 'DEFAULTNISTSP800131A'
   KEYS('leCTiYYS6fUPCgQsk5SHFBtTgADJHgax:gpfsRKMstanza')
  7. Test the new encryption policy.
    1. Create a file in the fs1 file system. 
      # echo 'Hello World!' > /fs1/testFile

      The policy engine detects the new file, encrypts it, and wraps the file encryption key in a master encryption key.

    2. To verify that the file /fs1/testFile is encrypted, issue the following command to display the encryption attribute of the file. 
       mmlsattr -n gpfs.Encryption /fs1/testFile
      The output shows that the file is encrypted:
      file name:            /fs1/testFile
gpfs.Encryption:      "EAGC????*Z????????????? ??????C>yA?????????????? ?2+]???ie???aq?q?Am?W0?Q??\?8?@???t??1UN?!?leCTiYYS6fUPCgQsk5SHFBtTgADJHgax?gpfsRKMstanza?"
EncPar 'AES:256:XTS:FEK:HMACSHA512'
        type: wrapped FEK  WrpPar 'AES:KWRAP'  CmbPar 'XORHMACSHA512'
                leCTiYYS6fUPCgQsk5SHFBtTgADJHgax:gpfsRKMstanza