Platform Manager certificate patch release notes

As platform manager certificates provided by IBM expire after 8 January 2022, you must update them before that date. Without applying this patch in time, you will not be able to run ap commands. After the patch is successfully applied, the certificates expiration date is extended to 100 years.

Normally, the certificates are updated automatically during upgrades. However, if the system was not upgraded for a long time, the certificates expiration date might be missed and cause the platform manager to stop working. You must run regenerate_certificates.py script to renew the Platform Manager certificates.

The patch also includes an update for the Call Home TrustList.jks file. The previously issued trust file expires in June 2022.
Note:

Running the patch is mandatory for any 1.0.x Cloud Pak for Data System. When you run the script, you can select not to update the certificates and only update the Call Home file, but it is not recommended.

There are two types of certificates, both of which must be upgraded:
  • REST certificates that are used for externally accessible REST API (for example: ap commands use this API).
  • Cluster certificates that are used for internal platform management communication, no endpoint accessible externally uses them.

Before you begin

  • The patch is applicable to any 1.0.x Cloud Pak for Data System version.
  • The estimated run time is 5-15 minutes, depending on the system size. Platform Manager is stopped, but the applications remain online. Database remains online.
  • The patch is executed by running an interactive regenerate_certificates.py script that guides you through the process.
  • If the certificate patch is applied and you want to upgrade to any later 1.0.x version, the platform management certificates (both, REST and cluster) are retained. However, Call Home TrustList.jks file gets overwritten in the upgrade process. You will have to rerun the script to update only the Call Home TrustList.jks file after the upgrade.
  • If your system has FIPS enabled, or SELinux is set to enforcing, you must disable FIPS and set SELinux to permissive before applying the patch. Upgrade does not preserve this configuration and fails if not disabled. apupgrade verifies this before upgrading. The settings must be re-enabled after the upgrade. For more information on these settings, see Configuring FIPS on pre-2.0.2 Cloud Pak for Data System and Configuring SELinux on Cloud Pak for Data System.

  • If your system has any non-NPS nodes disabled due to not being reachable, you can still run this script to apply certificates on other nodes. The script will report all unreachable nodes (along with NPS nodes) and prompt you to continue. To proceed, type y in 5. You will have to run the script again to apply certificates on these non-NPS unreachable nodes, after they become reachable.
Run the commands as root. Either log in as root directly, or use the command su -. The su root command does not work and causes the process to fail.

Procedure

  1. Download the 1.0.x.x.platform_management_certificate_patch.IF1-WS-ICPDS-fpxxx package from Fix Central.
  2. Once the tar file is downloaded, untar the file using tar -xvf command.
    tar -xvf certificates_regeneration-release-1.0.1.0-noarch.tar
    Example:
    [root@system localrepo]# tar -xvf certificates_regeneration-release-1.0.1.0-noarch.tar
    certificates_regeneration/
    certificates_regeneration/regenerate_certificates.py
    certificates_regeneration/TrustList.jks
    After the file is untared, the directory certificates_regeneration is created, containing both regenerate_certificates.py and TrustList.jks.
  3. Run:
    cd certificates_regeneration
  4. Run the following command without any parameters.
    python regenerate_certificates.py
  5. Wait for the nodes check to complete.
    Note: The script attempts to update the certificates on all nodes used in OpenShift. If NPS is installed, the NPS nodes are reported by the script as UNREACHABLE. Verify that only the NPS nodes are reported as unreachable, and, once prompted whether you want to continue, type y.
  6. When prompted to update the REST certificate:
    • Type y to update the REST certificate.
    • Type n not to update and press enter. Go to 8.
  7. When prompted, decide whether you want to apply your custom REST certificate (if you have it) or the default one:
    • Type y to provide the path to the custom REST certificate and key file. The custom REST certificate is applied.
    • Type n if you want to use the default REST certificate instead. The script then runs a default certificate expiry check and prompts you to type y to confirm the update. The default REST certificate is applied.
  8. When prompted to update the cluster certificates:
    • Type y to confirm. The cluster certificates are applied.
    • Type n not to update and press enter.
  9. The script automatically updates the TrustList.jks file. Wait for the script to start the platform manager and exit.
    Example from a system without NPS:
    [root@e1n1 ~]# python regenerate_certificates.py
    Started certificate regeneration script
    Checking nodes list...
    e1n1.fbond, e1n2.fbond, e1n3.fbond, e1n4.fbond, e2n1.fbond, e2n2.fbond, e2n3.fbond, e2n4.fbond
    Checking nodes reachability...
    Checking reachability of e1n1.fbond... ok
    Checking reachability of e1n2.fbond... ok
    Checking reachability of e1n3.fbond... ok
    Checking reachability of e1n4.fbond... ok
    Checking reachability of e2n1.fbond... ok
    Checking reachability of e2n2.fbond... ok
    Checking reachability of e2n3.fbond... ok
    Checking reachability of e2n4.fbond... ok
    All the nodes are reachable, proceeding
    Checking system state... Ready
    Checking for custom REST certificate... not found
    Checking default certificate expiry date...
    Default certificate expires on 2022-01-08 00:00:03 (in 87 days).
    Do you want to update it? y/[n]: y
    Certificate can be either regenerated automatically with default data or you can provide new custom one.
    Do you want to provide custom certificate? y/[n]: n
    Regenerating certificate automatically
    Generating new REST certificate in temp location... done
    Sending certificate to temporary locations on all the nodes
    Sending to e1n1.fbond... done
    Sending to e1n2.fbond... done
    Sending to e1n3.fbond... done
    Sending to e1n4.fbond... done
    Sending to e2n1.fbond... done
    Sending to e2n2.fbond... done
    Sending to e2n3.fbond... done
    Sending to e2n4.fbond... done
    Validating REST certificate on e1n1.fbond... ok
    Validating REST certificate on e1n2.fbond... ok
    Validating REST certificate on e1n3.fbond... ok
    Validating REST certificate on e1n4.fbond... ok
    Validating REST certificate on e2n1.fbond... ok
    Validating REST certificate on e2n2.fbond... ok
    Validating REST certificate on e2n3.fbond... ok
    Validating REST certificate on e2n4.fbond... ok
    Validation was successful on all the nodes
    Certificate for REST copied to nodes. Stopping platform management...
    Successfully deactivated platform
    Platform management stopped, applying new certificates
    Applying new certificates on e1n1.fbond... done
    Applying new certificates on e1n2.fbond... done
    Applying new certificates on e1n3.fbond... done
    Applying new certificates on e1n4.fbond... done
    Applying new certificates on e2n1.fbond... done
    Applying new certificates on e2n2.fbond... done
    Applying new certificates on e2n3.fbond... done
    Applying new certificates on e2n4.fbond... done
    REST certificate update was successful
    Checking for custom cluster certificates... not found
    Checking default certificates expiry date...
    Default certificates expire on 2022-11-19 17:06:12 (in 13 months).
    Cluster certificates can only be regenerated automatically, as they are internal ones, not exposed.
    Do you want to regenerate them? y/[n]: y
    Regenerating certificates
    Generating new cluster certificates in temp location... done
    Sending certificate to temporary locations on all the nodes
    Sending to e1n1.fbond... done
    Sending to e1n2.fbond... done
    Sending to e1n3.fbond... done
    Sending to e1n4.fbond... done
    Sending to e2n1.fbond... done
    Sending to e2n2.fbond... done
    Sending to e2n3.fbond... done
    Sending to e2n4.fbond... done
    Validating cluster certificates on e1n1.fbond... ok
    Validating cluster certificates on e1n2.fbond... ok
    Validating cluster certificates on e1n3.fbond... ok
    Validating cluster certificates on e1n4.fbond... ok
    Validating cluster certificates on e2n1.fbond... ok
    Validating cluster certificates on e2n2.fbond... ok
    Validating cluster certificates on e2n3.fbond... ok
    Validating cluster certificates on e2n4.fbond... ok
    Validation was successful on all the nodes
    Certificate for cluster copied to nodes. Stopping platform manager services...
    Platform management stopped, applying new certificates
    Applying new certificates on e1n1.fbond... done
    Applying new certificates on e1n2.fbond... done
    Applying new certificates on e1n3.fbond... done
    Applying new certificates on e1n4.fbond... done
    Applying new certificates on e2n1.fbond... done
    Applying new certificates on e2n2.fbond... done
    Applying new certificates on e2n3.fbond... done
    Applying new certificates on e2n4.fbond... done
    Cluster certificates update was successful
    Checking whether callhome container is running... yes
    Updating callhome trust list file...
    Backing up /opt/ibm/appliance/storage/platform/ras/callhome/ecc-root/config/TrustList.jks to /opt/ibm/appliance/storage/platform/ras/callhome/ecc-root/config/TrustList.jks.bak
    The TrustList.jks has been copied to /opt/ibm/appliance/storage/platform/ras/callhome/ecc-root/config/TrustList.jks
    Successfully updated, running apstart -p...
    Successfully activated platform
    Script is done, exiting
    [root@e1n1 ~]#
    
    Example from a system with NPS:
    [root@e1n1 ~]# python regenerate_certificates.py
    Started certificate regeneration script
    Checking nodes list...
    e1n1.fbond, e2n1.fbond, e3n1.fbond, e4n1.fbond, e5n1.fbond, e6n1.fbond
    Checking nodes reachability...
    Checking reachability of e1n1.fbond... ok
    Checking reachability of e2n1.fbond... ok
    Checking reachability of e3n1.fbond... ok
    Checking reachability of e4n1.fbond... ok
    Checking reachability of e5n1.fbond... UNREACHABLE
    Checking reachability of e6n1.fbond... UNREACHABLE
    Following nodes are unreachable (and won't be processed):
    e5n1.fbond, e6n1.fbond
    Continue? y/[n]: y
    Checking system state... Ready
    Checking for custom REST certificate... not found
    Checking default certificate expiry date...
    Default certificate expires on 2022-01-08 00:00:03 (in 3 months).
    Do you want to update it? y/[n]: y
    Certificate can be either regenerated automatically with default data or you can provide new custom one.
    Do you want to provide custom certificate? y/[n]: n
    Regenerating certificate automatically
    Generating new REST certificate in temp location... done
    Sending certificate to temporary locations on all the nodes
    Sending to e1n1.fbond... done
    Sending to e2n1.fbond... done
    Sending to e3n1.fbond... done
    Sending to e4n1.fbond... done
    Validating REST certificate on e1n1.fbond... ok
    Validating REST certificate on e2n1.fbond... ok
    Validating REST certificate on e3n1.fbond... ok
    Validating REST certificate on e4n1.fbond... ok
    Validation was successful on all the nodes
    Certificate for REST copied to nodes. Stopping platform management...
    Successfully deactivated platform
    Platform management stopped, applying new certificates
    Applying new certificates on e1n1.fbond... done
    Applying new certificates on e2n1.fbond... done
    Applying new certificates on e3n1.fbond... done
    Applying new certificates on e4n1.fbond... done
    REST certificate update was successful
    Checking for custom cluster certificates... not found
    Checking default certificates expiry date...
    Default certificates expire on 2022-11-19 17:06:12 (in 13 months).
    Cluster certificates can only be regenerated automatically, as they are internal ones, not exposed.
    Do you want to regenerate them? y/[n]: y
    Regenerating certificates
    Generating new cluster certificates in temp location... done
    Sending certificate to temporary locations on all the nodes
    Sending to e1n1.fbond... done
    Sending to e2n1.fbond... done
    Sending to e3n1.fbond... done
    Sending to e4n1.fbond... done
    Validating cluster certificates on e1n1.fbond... ok
    Validating cluster certificates on e2n1.fbond... ok
    Validating cluster certificates on e3n1.fbond... ok
    Validating cluster certificates on e4n1.fbond... ok
    Validation was successful on all the nodes
    Certificate for cluster copied to nodes. Stopping platform manager services...
    Platform management stopped, applying new certificates
    Applying new certificates on e1n1.fbond... done
    Applying new certificates on e2n1.fbond... done
    Applying new certificates on e3n1.fbond... done
    Applying new certificates on e4n1.fbond... done
    Cluster certificates update was successful
    Checking whether callhome container is running... yes
    Updating callhome trust list file...
    Backing up /opt/ibm/appliance/storage/platform/ras/callhome/ecc-root/config/TrustList.jks to /opt/ibm/appliance/storage/platform/ras/callhome/ecc-root/config/TrustList.jks.bak
    The TrustList.jks has been copied to /opt/ibm/appliance/storage/platform/ras/callhome/ecc-root/config/TrustList.jks
    Successfully updated, running apstart -p...
    Successfully activated platform
    Script is done, exiting
    [root@e1n1 ~]#
    
  10. Verify that certificates were properly updated by running the following commands and checking the Not After field:
    • REST certificates:
      openssl s_client -connect e1n1.fbond:5001 2>/dev/null | openssl x509 -text -noout
      Example:
      [root@e1n1 ~]# openssl s_client -connect e1n1.fbond:5001 2>/dev/null | openssl x509 -text -noout
      Certificate:
          Data:
              Version: 3 (0x2)
              Serial Number: 1 (0x1)
          Signature Algorithm: sha256WithRSAEncryption
              Issuer: C=US, CN=PlatformManager, ST=New York, L=Armonk, O=International Business Machines Corporation, OU=Analytics
              Validity
                  Not Before: Jan  1 00:00:00 1970 GMT
                  Not After : Sep 18 07:14:26 2121 GMT
              Subject: C=US, CN=PlatformManager, ST=New York, L=Armonk, O=International Business Machines Corporation, OU=Analytics
              Subject Public Key Info:
                      ...
      
    • Cluster certificates:
      openssl s_client -connect e1n1.fbond:5003 2>/dev/null | openssl x509 -text -noout
      Example:
      [root@e1n1 ~]# openssl s_client -connect e1n1.fbond:5003 2>/dev/null | openssl x509 -text -noout
      Certificate:
          Data:
              Version: 3 (0x2)
              Serial Number: 2 (0x2)
          Signature Algorithm: sha256WithRSAEncryption
              Issuer: C=US, CN=PlatformManager, ST=New York, L=Armonk, O=International Business Machines Corporation, OU=Analytics
              Validity
                  Not Before: Jan  1 00:00:00 1970 GMT
                  Not After : Sep 18 07:17:42 2121 GMT
              Subject: C=US, CN=PlatformManager, ST=New York, L=Armonk, O=International Business Machines Corporation, OU=Analytics
              Subject Public Key Info:
                      ...
      

Applying the patch on a system with expired certificates

If the certificates on your system already expired, you can still apply the patch. However, some workaround steps are required because Platform Manager is not fully operational in this state. After applying the procedure as described above, the system comes online, but there might be nodes marked disabled and Db2 is down (if db2wh service is installed). To overcome the situation, you need to run the following steps:

Procedure

  1. Run:
    apstop
  2. Run:
    apstart -p
    to start the platform manager only.
  3. Check if there are any disabled nodes by running:
    ap node
  4. Enable all disabled nodes found in 3 by running:
    ap node enable <node location>
  5. Run:
    apstart -a
    to start the system application.

Known issues

regenerate_certificates.py script fails with the following error: UnicodeEncodeError: 'ascii' codec can't encode character u'\u2018' in position 60: ordinal not in range(128)
Workaround:
  1. Add the following lines:
    reload(sys)
    sys.setdefaultencoding('UTF8')
    in regenerate_certificates.py script after the import sys line.
  2. Rerun the script.