Node identities

Creating a cluster with node identities

This procedure demonstrates how to create a multi-node cluster with proper TLS certificate configuration. You will create a CA, generate node-specific certificates with SANs, and configure the cluster to use these identities for secure communication.

All nodes in a IBM Storage Scale cluster must have a TLS certificate with the CN set to the cluster name. To support multiple nodes in the cluster, use SANs in each certificate to identify the node hostnames.

For example, create the IBM Storage Scale cluster scale.cluster with the nodes node1.example.net and node2.example.net.

1. Create a directory (tls) in the home folder and navigate to that directory for storing certificate files.

   mkdir -p ~/tls
   cd ~/tls

2. Generate a private key (ca.key) for the CA.

   openssl ecparam -name prime256v1 -genkey -noout -out ca.key

3. Create a self-signed CA certificate (ca.crt) to sign the node certificates as a trusted root.

   openssl req -new -x509 -sha256 -key ca.key -out ca.crt -subj "/O=YourCompany/CN=ScaleCA" -days 10000

   Note: Replace YourCompany with your organization name.
4. Set the node name as a shell variable and create a node identity for node1.example.net.

   NODENAME=node1.example.net
   openssl ecparam -name prime256v1 -genkey -noout -out ${NODENAME}.key
   openssl req -new -sha256 -key ${NODENAME}.key -out ${NODENAME}.csr -subj "/O=YourCompany/CN=scale.cluster"

5. Create a SAN configuration file (san.conf) for node1.example.net that includes the short node name and the FQDN.

   cat << EOF > san.conf
   
   [ v3_req ]
   
   subjectAltName = @alt_names
   
   [ alt_names ]
   
   DNS.1 = ${NODENAME%%.*}
   
   DNS.2 = ${NODENAME}
   
   EOF

6. Generate a signed node certificate for node1.example.net with the self-signed CA certificate and the SAN configuration file.

   openssl x509 -req -in ${NODENAME}.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out ${NODENAME}.pem -days 10000 -sha256 -extensions v3_req -extfile san.conf

7. Verify the signed node certificate for node1.example.net.

   openssl verify -CAfile ca.crt ${NODENAME}.pem

   A successful output displays: ${NODENAME}.pem: OK
   Note: If certificate verification fails, ensure:

   • The CA certificate (ca.crt) is the same one used to sign the node certificate.
   • The certificate files have correct permissions (readable by the user).
   • The certificate has not expired.
8. Set the node name variable and create a node identity for node2.example.net.

   NODENAME=node2.example.net
   openssl ecparam -name prime256v1 -genkey -noout -out ${NODENAME}.key
   openssl req -new -sha256 -key ${NODENAME}.key -out ${NODENAME}.csr -subj "/O=YourCompany/CN=scale.cluster"

9. Create a SAN configuration file (san.conf) for node2.example.net that includes the short node name and the FQDN.

   cat << EOF > san.conf
   
   [ v3_req ]
   
   subjectAltName = @alt_names
   
   [ alt_names ]
   
   DNS.1 = ${NODENAME%%.*}
   
   DNS.2 = ${NODENAME}
   
   EOF

10. Generate a signed node certificate for node2.example.net with the self-signed CA certificate and the SAN configuration file.

    openssl x509 -req -in ${NODENAME}.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out ${NODENAME}.pem -days 10000 -sha256 -extensions v3_req -extfile san.conf

11. Verify the signed node certificate for node2.example.net.

    openssl verify -CAfile ca.crt ${NODENAME}.pem

    A successful output displays: ${NODENAME}.pem: OK
12. Copy the certificate files to each node. Ensure the following files are available on each node:
    • Node-specific certificate: node1.example.net.pem or node2.example.net.pem
    • Node-specific private key: node1.example.net.key or node2.example.net.key
    • CA certificate: ca.crt (same file on all nodes)
    Note: Use scp or another secure file transfer method to copy files between nodes.
13. Import the generated node identities on each node. You must run this command locally on each node before creating the cluster.
    On node1:

    scalectl node config set --cert ~/tls/node1.example.net.pem --key ~/tls/node1.example.net.key --chain ~/tls/ca.crt

    On node2:

    scalectl node config set --cert ~/tls/node2.example.net.pem --key ~/tls/node2.example.net.key --chain ~/tls/ca.crt

14. To allow scalectl and other client commands to validate the server certificate, add the CA certificate to the system trust store.

    cp ~/tls/ca.crt /etc/pki/ca-trust/source/anchors
    update-ca-trust

15. Create the cluster and add the additional node.

    scalectl cluster create --node-name node1 --cluster-name scale.cluster
    mmchlicense server --accept -N all
    scalectl node add -N node2

SAN configuration examples

tls: failed to verify certificate: x509: cannot validate certificate for 192.168.1.10 because it doesn't contain any IP SANs:
make sure --url flag target matches the SAN addresses of the Admin Daemon's node identity

A SAN configuration file can include DNS names, wildcard domains, and IP addresses. A sample san.conf file is as follows:

[ v3_req ]
subjectAltName = @alt_names

[ alt_names ]
DNS.1 = example.com
DNS.2 = www.example.com
DNS.3 = api.example.com
DNS.4 = *.internal.example.com
IP.1  = 192.168.1.10
IP.2  = 10.0.0.5
IP.3  = 127.0.0.1

openssl x509 -noout -text -in server.pem

Configure CA trust chain

tls: failed to verify certificate: x509: certificate signed by unknown authority: unable to validate Admin Daemon's node identity. Use the --cacert flag to point to a valid CA chain to validate the server certificate, 
add the CA chain to your system trust store, or disable client side validation using --insecure-skip-tls-verify flag

Method 1: Add the CA to the system trust store

This method is operating system (OS) specific. For example, on RHEL 9.5:

Attention: Adding a CA to the system trust store applies globally to all TLS connections. Only use this approach if you understand the associated security implications.

1. Copy the CA certificate to the system anchor directory.

   cp ~/tls/ca.crt /etc/pki/ca-trust/source/anchors

2. Update the systems trust store to include the newly added CA certificates to the anchor directory.

   update-ca-trust

Method 2: Use the --cacert flag with scalectl

In this method, the CA certificate that is to be trusted is specified for each scalectl command. This method does not require the CA certificate to be trusted globally for a system but allows clients to validate the server certificate. It is equivalent to using the --cacert flag with curl.

Node identity import reference

This section provides reference information about importing node identities. For a complete procedure that includes certificate generation and cluster creation, see Creating a cluster with node identities.

When nodes communicate, both the initiating (client) and receiving (server) nodes must authenticate each other. Node identities must be configured in an all-to-all manner so that each node can act as both a client and a server. IBM Storage Scale uses TLS version 3 (TLSv3) for secure communication between nodes.

scalectl node config set --cert <path to certificate> --key <path to private key> --chain <path to ca chain>