Enabling SSL for the RESTful APIs using a self-signed certificate

Configure SSL for the web servers hosting the RESTful APIs, using a self-signed certificate for testing purposes.

Before you begin

You must be a cluster administrator.
You must ensure that the clock settings on the server and client are identical. Otherwise, errors may occur.

About this task

Follow these steps if you want to:

Enable SSL for the first time (in other words, if you disabled SSL during installation by setting DISABLESSL=Y).
Use your own self-signed certificate that will replace the default one generated when SSL is enabled during installation.

Configure SSL for the following web servers to safely access the RESTful APIs:

The ascd web server, which hosts the RESTful APIs for Spark instance group and application instance management.
The REST web server, which hosts the RESTful APIs for resource management and package deployment.

When SSL is enabled for the web services (ascd and REST), a trust relationship between the server and the client is established by sending a server certificate to the client. The client validates the certificates that are signed by the self-signed Platform Computing CA Root. This self-signed certificate can be used only for testing purposes. For your production environment, use a properly chained certificate that is issued or signed by a trusted certificate authority. By default, ascd and REST uses the TLSv1.2 protocol.

Important: You must use the same SSL setting for the cluster management console and the RESTful web servers. If you enable SSL for one, you must also enable SSL for the other; if you disable SSL for one, SSL must be disabled for the other as well. This setting also takes effect for cloud bursting with host factory. Ensure that SSL for all these functions is configured consistently in the cluster; without a uniform configuration, errors occur. Note, however, that when SSL is uniformly enabled, you can use different certificates and keys as required.

Note: On a local file system, Tier 1 files must be generated on all management hosts. On a shared file system, the files need to be generated on a shared file system location that all management hosts can access. Also, the subjectAltName (SAN) field must contain all management hosts.

Procedure

Stop the service:
- To stop the ascd service, run:
```
egosh service stop ascd 
```
- To stop the REST service, run:
```
egosh service stop REST
```
If you want to manage your own certificates, use the keytool utility to clean up the certificate in the default server keystore (serverKeyStore.jks):

Note: This step is required only if SSL was previously enabled (thus, generating the serverKeyStore.jks). In this case, you must clean up the certificate.
1. Back up the default server keystore ($EGO_TOP/wlp/usr/shared/resources/security/serverKeyStore.jks).
2. Change to the /bin directory under your Java™ directory:
```
cd $EGO_TOP/jre/8.0.5.25/linux-x86_64/bin
```
3. List the certificates in the keystore and identify its alias. For example:
```
keytool -v -list -keystore serverKeyStore.jks -storepass storepass
```
  where storepass specifies the keystore password; for example: Liberty.
  In the following example, the generic domain certificate uses the alias srvalias:
```
   ...
    Alias name: srvalias
    Creation date: Nov 5, 2015
    Entry type: keyEntry    
    Certificate chain length: 2
    ...
```
4. Remove the certificate from the keystore. For example:
```
keytool -delete -alias srvalias -keystore serverKeyStore.jks -storepass Liberty
```
5. To remove the Root certificate in the keystore, list the certificates to look for the Root certificate. Look for the one whose Owner and Issuer has the value of CN=IBM Spectrum Computing Root CA, O=IBM, C=CA.
```
keytool -v -list -keystore serverKeyStore.jks -storepass storepass
```
  where storepass specifies the keystore password; for example: Liberty.
  In the following example, the external certificate uses the alias caalias:
```
   ...
    Alias name: caalias
    Creation date: Nov 5, 2015
    Entry type: trustedCertEntry
    
    Owner: CN=IBM Spectrum Computing Root CA, O=IBM, C=CA    
    Issuer: CN=IBM Spectrum Computing Root CA, O=IBM, C=CA
    ...
```
6. Delete the Root certificate from the keystore
```
keytool -delete -alias caalias -keystore serverKeyStore.jks -storepass storepass
```
  where storepass specifies the keystore password; for example: Liberty.
Generate a self-signed certificate using the keytool utility:

Important: When you run keytool, the server keystore is generated in a local directory, not in a shared directory. If the ascd and REST services are configured to use the ManagementHosts resource group or you have customized the host list, ensure that you complete these steps on all management hosts, so that the keystore is available on all management hosts.
1. From the /bin directory under your Java directory, generate a key pair for the server keystore:
```
keytool -genkeypair -noprompt -alias srvalias -dname "CN=*.domainName,O=IBM,C=CA" -keystore serverKeyStore.jks 
-storepass storepass -keypass keypass -keyalg rsa -validity 1095 -keysize  2048 -sigalg SHA256withRSA 
-ext "san=dns:hostname"
```
  where:
  - -dname domainName identifies the fully qualified domain of the web server. For example, if the web server host name is ascdrestserver.example.com, then the domainName value is example.com. Ensure that the domain name does not start with a number.
  - -storepass storepass specifies the password to protect the integrity of the keystore (for example: Liberty) and -keypass keypass specifies the password for the keystore (for example: Liberty). It is recommended that you use the same password for both options.
  - -alias is set to use srvalias.
  - -ext "san=dns:hostname" specifies all management host names to which the certificate must apply as the subjectAltName (SAN). The master host that is defined during installation is automatically added as the subjectAltName. If you installed your cluster to a shared file system and enabled master host failover, add the names of all your master-candidate hosts (-ext "san=dns:hostname1,dns:hostname2,...").
    Note: If you add primary-candidates after the certificate is first generated, you must generate a new certificate to include the host names of the newly added primary-candidates.
2. Create the certificate-signing request (.csr file):
```
keytool -certreq -alias srvalias -file srvcertreq.csr -storepass Liberty -keystore serverKeyStore.jks 
-ext "san=dns:hostname"
```
3. Retrieve and save the following CA key and the CA root certificates:
  - $EGO_TOP/wlp/usr/shared/resources/security/cacert.pem
  - $EGO_TOP/wlp/usr/shared/resources/security/caKeyStore.jks
4. Sign the certificate using the CA key:
```
keytool -gencert -infile srvcertreq.csr -outfile servercertcasigned.pem -alias caalias -keystore caKeyStore.jks_location 
-storepass storepass -validity 1095 -keysize 2048 -sigalg SHA256withRSA -keyalg rsa -ext "san=dns:hostname"
```
  where:
  - caKeyStore.jks_location is the path to the caKeyStore.jks file that you saved previously.
  - storepass specifies the keystore password; for example: Liberty.
5. Import the CA root certificate to the server keystore:
```
keytool -importcert -noprompt -alias caalias -file cacert.pem_location -keystore serverKeyStore.jks -storepass storepass
```
  where:
  - cacert.pem_location is the path to the cacert.pem file that you saved previously.
  - storepass specifies the keystore password; for example: Liberty.
6. Import the signed certificate back into the keystore:
```
keytool -importcert -noprompt -alias srvalias -file servercertcasigned.pem -storepass storepass -keystore serverKeyStore.jks
```
  where storepass specifies the keystore password; for example: Liberty.
Copy the serverKeyStore.jks file to the security directory: $EGO_TOP/wlp/usr/shared/resources/security/
Configure SSL settings in the configuration file:
1. Copy the SSL settings from the *.xml.ssl file:
  - For ascd, copy $EGO_CONFDIR/../../ascd/conf/server_ascd.xml.ssl to $EGO_CONFDIR/../../ascd/conf/server_ascd.xml.
  - For REST, copy $EGO_CONFDIR/server_rest.xml.ssl to $EGO_CONFDIR/server_rest.xml.
2. Open the configuration file:
  - For ascd, open $EGO_CONFDIR/../../ascd/conf/server_ascd.xml.
  - For REST, open $EGO_CONFDIR/server_rest.xml.
3. Define the default keystore with the password and its location in the keyStore element.
```
<keyStore id ="defaultKeyStore" password="storepass" location="${wlp.user.dir}/shared/resources/security/serverKeyStore.jks"/>
```
  where storepass specifies the keystore password; for example: Liberty.
4. Optional: Encrypt the password using the securityUtilities command. For more information, see:
  - Liberty profile: securityUtility command
5. Save your changes to server_ascd.xml or server_rest.xml.
If you are enabling SSL for the first time or you are using custom certificates, then you need to copy the CA public certificate to the same location on all hosts.
1. Edit the CA_CERT_KEYSTORE_PASSWD, CA_CERT_KEYSTORE_PATH, and CA_CERT_PATH parameters in $EGO_CONFDIR/../../ascd/conf/ascd.conf. For example,
```
CA_CERT_KEYSTORE_PASSWD={aes}ANr2+3vBND5lpzCl4QWrolYXCy3OggZWgvuDsjD7c5vF
CA_CERT_KEYSTORE_PATH=$EGO_TOP/security/caKeyStore.jks
CA_CERT_PATH=$EGO_TOP/wlp/usr/shared/resources/security/cacert.pem
```
2. Double check the value of the CA_CERT_PATH parameter in $EGO_CONFDIR/../../integration/elk/conf/elk.conf.
  By default this value is set to CA_CERT_PATH=${EGO_TOP}/wlp/usr/shared/resources/security/cacert.pem.
3. Restart the following services if you changed the value of CA_CERT_PATH in elk.conf:
  - elk-manager
  - elk-elasticsearch
  - elk-elasticsearch-master
  - elk-elasticsearch-data
For the ascd service with application container workload, if you want to manage application instances from the command line (using the egosh appinstance command) with SSL, you must define the following environment variables for the self-signed certificate to avoid untrusted connection issues:
- EGO_SSL_EGOSH_CAINFO: Specifies the path to the default self-signed certificate, which is available at $EGO_TOP/wlp/usr/shared/resources/security/cacert.pem.
- EGO_SSL_EGOSH_VERIFYHOST: Enables or disables verification of the Common Name (CN) field in the certificate against the host name (displayed in the Description field of the egosh client view ASCD_REST_BASE_URL_si_seqno command). If the CN field in the CA-signed certificate does not match the host name or if you want to avoid untrusted connection issues when using the default self-signed certificate, set this variable to N.
- EGO_SSL_EGOSH_VERIFYPEER: Enables or disables verification of the peer SSL certificate. To disable verification of the peer SSL certificate or if you want to avoid untrusted connection issues when using the default self-signed certificate, set this variable to N.
For Linux® hosts, these variables are also defined by default in the environment profile scripts ($EGO_TOP/ascd/conf/profile.ascd and $EGO_TOP/ascd/conf/cshrc.ascd). Change the settings using the scripts, if required.
For more information on the environment variables, see appinstance.
Start the service:
- To start the ascd service, run:
```
egosh service start ascd 
```
- To start the REST service, run:
```
egosh service start REST
```
Restart all Spark instance groups in the cluster.
Update the following environment variables that are used by the SparkCleanup service:
- ASCD_REST_URL: Defines the base URL for the ascd web server and is required by the SparkCleanup service to periodically clean up the environment for all Spark instance groups in your cluster.
- ASCD_REST_CACERT_PATH: If you are enabling SSL for the first time or you are using custom certificates, you need to edit this environment variable to the path that you specified in step 6.
1. From the cluster management console, select System & Services > EGO Services > Service Profiles.
2. In the Other Services tab, click SparkCleanup.
3. In the service profile dialog, edit the ASCD_REST_URL and the ASCD_REST_CACERT_PATH environment variables under sc:ActivityDescription > ego:ActivitySpecification. For example:
  - For ASCD_REST_URL with SSL enabled, the default URL is https://${SERVICE_ascd_LOCATION}:8643/platform/rest.
  - For ASCD_REST_CACERT_PATH: $EGO_TOP/wlp/usr/shared/resources/security/cacert.pem
4. Click Save.
Launch a browser to access the RESTful APIs over SSL.
- To access the APIs for application instance management, enter (for example) https://ascrestserver.example.com:8643/platform/rest/asc/v1.
  If you set EGO_SSL_EGOSH_VERIFYHOST and EGO_SSL_EGOSH_VERIFYPEER to N, you can access the RESTful APIs. Otherwise, continue with the rest of this procedure.
- To access the APIs for Spark instance group management, enter (for example) https://ascrestserver.example.com:8643/platform/rest/conductor/v1.
- To access the APIs for resource management, enter (for example) https://restserver.example.com:8543/platform/rest/ego/v1.
- To access the APIs for package deployment, enter (for example) https://restserver.example.com:8543/platform/rest/deployment/v1.
When the browser prompts you about an untrusted connection, download and import the web server certificate:
1. With the appropriate permissions and access, obtain the certificate from the server file system at $INSTALL_TOP/wlp/usr/shared/resources/security/cacert.pem.
2. Configure your browser:
  The following steps apply only if you are using the default self-signed certificate with the certified browsers; the steps might be different with other browser versions.
  - If you are using Chrome on Windows, complete the following steps:
    1. Click Settings > Advanced > Privacy and Security > Manage Certificates > Import.
    2. Follow the Certificate Import Wizard, import the downloaded cacert.pem, click Place all certificates in the following store, click Browse, select Trusted Root Certification Authorities, and click OK.
    3. Complete the wizard and accept any additional prompts.
    4. Restart the browser, if required.
  - If you are using Chrome on Mac OS, complete the following steps. You are required to enter your Keychain password for some of these steps:
    1. Double-click the cacert.pem file in the Finder.
    2. In the Add Certificates window, select System from the Keychain drop-down menu.
    3. Click Add.
    4. In the Keychain Access window, select the System tab and double-click the imported certificate.
    5. Under Trust, select Always Trust for all of the drop-down options.
    6. Exit the Keychain and restart the browser.
  - If you are using Firefox on Windows or Linux, complete the following steps:
    1. Click Tools > Options > Advanced.
    2. On the Certificates tab, click View Certificates.
    3. On the Authorities tab, click Import.
    4. Select the certificate from the location where it is saved on your system.
    5. When prompted to trust a new CA, ensure that you select Trust this CA to identify websites and click OK.
    6. Restart the browser, if required.
  - If you are using Firefox on Mac OS, complete the following steps:
    1. Click Firefox > Preferences > Advanced > View Certificates.
    2. Click the Authorities tab and click Import.
    3. Select the certificate from the location where it is saved on your system.
    4. When prompted to trust a new CA, ensure that you select Trust this CA to identify websites and click OK.
    5. Restart the browser, if required.

Results

You can now securely access the RESTful APIs over SSL in your development environment.

What to do next

If the cluster contains Spark instance groups with notebooks or Spark instance groups that are configured to use GPU executors and have the SPARK_EGO_AUTOSCALE_SLOTS_PER_TASK parameter set, you or the consumer administrator must modify these Spark instance groups so that ascd can apply the appropriate service profile changes. To modify, stop the Spark instance group, and click Configure > Modify Instance Group.