Setting up SSL for instance groups: Tier 2 and Tier 3 (self-signed certificates)

Set up SSL for the cluster for tier 2 services and tier 3 Spark drivers so that SSL is turned on by default for instance groups; using a self-signed certificate for testing purposes.

Before you begin

To manually enable SSL, you must ensure that:

You must be a cluster administrator.
You must enable tier 1 SSL files and configuration for the cluster management console and RESTful APIs. See Enabling SSL for the cluster management console using a self-signed certificate and Enabling SSL for the RESTful APIs using a self-signed certificate.

Alternatively, to automatically enable SSL, see Enabling SSL with the SSL tool.

About this task

Follow this task if you want to:

Enable SSL for the first time for instance groups (in other words, if you disabled SSL during installation by setting DISABLESSL=Y).
To use your own self-signed certificate that replaces the default one generated when SSL is enabled during installation.

Note: On a local file system, Tier 2 and 3 files must be generated on all hosts in the cluster. On a shared file system, the files need to be generated on a shared file system location that all hosts in the cluster can access. Also, the subjectAltName (SAN) field must contain all hosts in the cluster.

Procedure

If you want to manage your own certificates, use the keytool utility to clean up the certificate in the default server keystore (tier2and3ServerKeyStore.jks):

Note: This step is required only if SSL was previously enabled (thus, generating the tier2and3ServerKeyStore.jks). In this case, you must clean up the certificates.
1. Back up the default server keystore ($EGO_TOP/security/tier2and3ServerKeyStore.jks).
2. Change to the /bin directory under your Java™ directory:
  For example:
  - ```
  cd $EGO_TOP/jre/8.0.6.0/linux-x86_64/bin
```
- ```
cd $EGO_TOP/jre/8.0.6.0/linux-ppc64le/bin
```
3. List the certificates for tier 2 and tier 3 in the keystore and identify the alias'.
```
keytool -v -list -keystore $EGO_TOP/security/tier2and3ServerKeyStore.jks -storepass storepass
```
  where storepass specifies the keystore password. The default password is SparkPassword.
  In the following example, the generic domain certificates use the alias' tier2alias and tier3alias:
```
   ...
    Alias name: tier2alias
    Creation date: Nov 5, 2015
    Entry type: keyEntry    
    Certificate chain length: 2
   ...
```
```
   ...
    Alias name: tier3alias
    Creation date: Nov 5, 2015
    Entry type: keyEntry    
    Certificate chain length: 2
   ...
```
4. Remove the certificate from the keystore for tier 2 and tier 3.
  - Tier 2 example:
```
keytool -delete -alias tier2alias -keystore $EGO_TOP/security/tier2and3ServerKeyStore.jks -storepass storepass
```
    where storepass specifies the keystore password.
  - Tier 3 example:
```
keytool -delete -alias tier3alias -keystore $EGO_TOP/security/tier2and3ServerKeyStore.jks -storepass storepass
```
5. To remove the Root certificate in the keystore for tier 2 and tier 3, list the certificates to look for the Root certificate. Look for the one whose Owner and Issuer has the value of CN=IBM Spectrum Computing Root CA, O=IBM, C=CA.
  - For example:
```
keytool -v -list -keystore $EGO_TOP/security/tier2and3ServerKeyStore.jks -storepass storepass
```
    where storepass specifies the keystore password.
    In the following example, the external certificate uses the alias caalias:
    ... Alias name: caalias Creation date: Nov 5, 2015 Entry type: trustedCertEntry Owner: CN=IBM Spectrum Computing Root CA, O=IBM, C=CA Issuer: CN=IBM Spectrum Computing Root CA, O=IBM, C=CA ...
6. Delete the Root certificate from the keystore for tier 2 and tier 3:
  - For example:
```
keytool -delete -alias caalias -keystore $EGO_TOP/security/tier2and3ServerKeyStore.jks -storepass storepass
```
    where storepass specifies the keystore password.
Generate one keystore for tier 2 and tier 3 using the keytool utility:
From the /bin directory under your Java directory, run the following command to generate a keystore:
- Tier 2 example:
```
keytool -genkeypair -noprompt -alias tier2alias -dname "CN=*.domainName,O=IBM,C=CA" -keystore $EGO_TOP/security/keystore_name.jks  
-storepass storepass -keypass keypass -keyalg rsa -validity 825 -keysize  2048 -sigalg SHA256withRSA 
-ext "san=dns:hostname"
```
  where:
  - -alias is set to use tier2alias.
  - -dname domainName identifies the fully qualified domain of the cluster management console web server. For example, if the web server host name is hostname.example.com then the domainName value is example.com.
  - keystore_name is the name of your new keystore. You should put the keystore in a secure location that will not be deleted, for example $EGO_TOP/security/. You can reuse the default server keystore (tier2and3ServerKeyStore.jks) after cleanup (see step 1).
  - -storepass storepass specifies the password to protect the integrity of the keystore (for example: storepass) and -keypass keypass specifies the password for the alias. It is recommended that you do not use the same password for both options.
  - -ext "san=dns:hostname" specifies all host names in the cluster to which the certificate must apply as the subjectAltName. For example, -ext "san=dns:hostname1,dns:hostname2,...".
- Tier 3 example:
```
keytool -genkeypair -noprompt -alias tier3alias -dname "CN=*.domainName,O=IBM,C=CA" -keystore keystore_name.jks  
-storepass storepass -keypass keypass -keyalg rsa -validity 825 -keysize  2048 -sigalg SHA256withRSA 
-ext "san=dns:hostname"
```

Create the certificate-signing request file (.csr file) for tier 2 and tier 3:

Tier 2 example:
```
keytool -certreq -alias tier2alias -file tier2alias.csr -keystore keystore_name.jks -storepass storepass -keypass keypass 
```
where tier2alias.csr is the certificate-signing request file that you require to order an SSL certificate from your CA vendor.

Tier 3 example:

keytool -certreq -alias tier3alias -file tier3alias.csr -keystore keystore_name.jks -storepass storepass -keypass keypass

Retrieve and save the following CA key and the CA root certificates:
- $EGO_TOP/wlp/usr/shared/resources/security/cacert.pem
- $EGO_TOP/wlp/usr/shared/resources/security/caKeyStore.jks
Sign the certificate by using the CA key for tier 2 and tier 3:
- Tier 2 example:
```
keytool -gencert -infile tier2alias.csr -outfile tier2aliascertcasigned.pem -alias caalias -keystore caKeyStore.jks_location 
-storepass Liberty -validity 825 -keysize 2048 -sigalg SHA256withRSA -keyalg rsa 
-ext "san=dns:hostname"
```
  where:
  - caKeyStore.jks_location is the path to the caKeyStore.jks file that you saved previously.
  - Liberty specifies the keystore password.
  - -ext "san=dns:hostname" specifies the host name to which the certificate must apply as the subjectAltName. The primary host that is defined during installation is automatically added as the SubjectAltName. If you installed your cluster to a shared file system and enabled primary host failover, add the names of all your primary-candidate hosts (-ext "san=dns:hostname1,dns:hostname2,...").
- Tier 3 example:
```
keytool -gencert -infile tier3alias.csr -outfile tier3aliascertcasigned.pem -alias caalias -keystore caKeyStore.jks_location 
-storepass Liberty -validity 825 -keysize 2048 -sigalg SHA256withRSA -keyalg rsa 
-ext "san=dns:hostname"
```
Import the CA root certificate to the server keystore:
For example:
```
keytool -importcert -noprompt -alias caalias -file cacert.pem_location -keystore $EGO_TOP/security/keystore_name.jks -storepass storepass 
```
where:
- $EGO_TOP/security/keystore_name.jks is the path to the keystore that was previously created.
- cacert.pem_location is the path to the cacert.pem file that you saved previously.
- storepass specifies the keystore password.
Import the signed certificate back into the keystore:
- Tier 2 example:
```
keytool  -import -noprompt -alias tier2alias -file tier2aliascertcasigned.pem -storepass storepass -keypass keypass -keystore $EGO_TOP/security/keystore_name.jks
```
  where
  - $EGO_TOP/security/keystore_name.jks is the path to the keystore that was previously created.
  - storepass specifies the keystore password; for example: SparkPassword.
  - keypass specifies the password for the alias. It is recommended that you do not use the same password for storepass and keypass.
- Tier 3 example:
```
keytool  -import -noprompt -alias tier3alias -file tier3aliascertcasigned.pem -storepass storepass -keypass keypass -keystore $EGO_TOP/security/keystore_name.jks
```
Copy keystore_name.jks with 444 permissions on all hosts in the cluster.
Optional: Encrypt the passwords that you specified in the previous steps by using the securityUtilities command. For more information, see:
- Creating SSL certificates from the command line.
- Liberty profile: securityUtility command
Edit $EGO_CONFDIR/../../ascd/conf/ssl.conf to modify the following parameters to the values you used in the previous steps:
```
KEYSTORE_PATH=keystore_name.jks
KEYSTORE_PASSWD={aes}keystore_password
KEYSTORE_TYPE=JKS
TIER2_ALIAS_NAME=tier2AliasName
TIER2_KEY_PASSWD={aes}password_tier2
TIER3_ALIAS_NAME=tier3AliasName
TIER3_KEY_PASSWD={aes}password_tier3
```
Passwords are encrypted by $EGO_TOP/wlp/$WLP_VERSION/bin/securityUtility with AES (if you did that step/topic):
Optional: If the instance groups support notebooks, you can generate your own open SSL certificates from your keystore files, see Setting up SSL for instance groups: Tier 3 notebooks.
Edit $EGO_CONFDIR/../../ascd/conf/ascd.conf to turn SSL on by default for instance groups and to provide the Spark services with the required keystore information:
```
CONDUCTOR_SPARK_SSL_ENABLED=ON
CA_CERT_KEYSTORE_PATH=$EGO_TOP/wlp/usr/shared/resources/security/caKeyStore.jks
CA_CERT_KEYSTORE_PASSWD={aes}encodedpassword 
```
Restart ascd to accept the changes to the properties file:
1. Stop the ascd service by running the following command:
```
egosh service stop ascd
```
1. Start the ascd service by running the following command:
```
egosh service start ascd
```

Results

SSL is now set up for instance groups in the cluster. SSL can be enabled for newly created instance groups by using the cluster management console. See Enabling SSL for the instance group. Existing instance groups can be modified to enable SSL. See Modifying instance groups