Flashes (Alerts)
Abstract
IBM Spectrum Scale Encryption uses certificates to authenticate a connection between a key client and a key server. A key client certificate and key server certificate will expire some time after configuring encryption, resulting in loss of access to the encrypted files.
Content
If a file system is encrypted, the GPFS daemon acts as a key client and requests master encryption keys (MEKs) from a key server. Authentication between the key client and the key server requires valid certificates that are not expired. When the key client or the key server certificate expires, the IBM Spectrum Scale client can no longer mount the encrypted file system or access encrypted files, because it can no longer retrieve MEKs from the key server. To restore file access to encrypted files, the expired certificate(s) must be renewed, and the IBM Spectrum Scale or the key server must trust the new certificate(s).
Users Affected:
This issue affects customers running IBM Spectrum Scale V5.0.x (ESS V5.3.x and ESS V 6.0.x), and V5.1.0 code levels when IBM Spectrum Scale encryption feature is used and the key client or the key server certificate expires. For more information about IBM Spectrum Scale Encryption feature, see the Encryption chapter:
Problem Determination:
- When the certificate of a key server expires, any attempt to create, open, read, or write encrypted files fail with an "Operation not permitted" error. The following error message is logged in the daemon log file( /var/adm/ras/mmfs.log.latest):
[W] The key server sklm1 (port 5696) had a failure and will be quarantined for 1 minute(s).
[E] Unable to create encrypted file file.enc (inode 21260, fileset 0, file system gpfs).
[E] Key 'KEY-uuid:sklm1' could not be fetched. Bad certificate.
- When IBM Spectrum Scale detects an expired client certificate, the following error message is logged in the daemon log file (/var/adm/ras/mmfs.log.latest):
[E] Failed to open expire.
[W] Command: err 666: mount gpfs
[E] Operation not permitted
[E] mount: mount expire on /gpfs failed: Stale file handle
- The simplified setup doesn't support the use of a server certificate chain.
- When IBM Spectrum Scale client detects an expired key client certificate, it prints the following error message.
[E] Error while validating policy 'for file system gpfs': rc=778: While parsing file '/var/mmfs/etc/RKM.conf':GPFS: 6027-3535 [E]
Incorrect client certificate label 'client1' for backend 'RKM1'.
This message was fixed in IBM Spectrum Scale 5.0.0.0 or later to state that the client certificate has expired.
Recommendations:
Identify certificates that are approaching their expiration date:
-
In IBM Spectrum Scale V 5.0.2 (ESS 5.3.2) or later, the GPFS daemon writes warning messages into the daemon log file (/var/adm/ras/mmfs.log.latest) for certificates that are nearing their expiration dates. Warnings are issued for both key client and key server certificates.
-
Key server expiring certificate warning example:
The warning message for a key server certificate that is approaching its expiration date contains the date and time of expiration, and the IP address and port of the key server, as shown in the following example:[W] The server certificate for key server 192.168.9.135 (port 5696) will expire at Nov 10 12:03:32 2020 EDT (-0400).
-
Key client expiring certificate warning example:
The warning message for a key client certificate that is approaching its expiration date contains the date and time of the expiration, the IP address and port of the key server to which the key client has a connection, the label of the client certificate and the RKM stanza name. For example:[W] The client certificate with label 'client1' for key server with RKM ID 'RKM1' (192.168.9.123:5696) will expire at Nov 20 16:39:59 2020 EDT (-0400).
For more information, see Certificate expiration warnings section:
- View and check the expiration date of a key client and a key server certificate depending on the IBM Spectrum Scale version installed and the encryption setup method used to configure encryption. For more information, see Certificate expiration dates and error messages section:
-
Renew an expired client certificate for the simplified setup and the regular setup. Follow the steps in the "Renewing expired client certificates" section, in the "Encryption" chapter:
-
Renew an expired server certificate for the simplified setup, and the regular setup. Follow the steps in "Renewing expired server certificates" section, in the "Encryption" chapter:
Was this topic helpful?
Document Information
Modified date:
19 November 2020
UID
ibm16369269