Automating the distribution of IBM Storage Protect server certificate to clients

Self-signed SSL/TLS certificate that is generated by the IBM Storage Protect server expires after 10 years. You can renew the certificate when it is expired or before it expires. Here an utility is provided, which helps automate the distribution of new certificate generated on IBM®® Storage Protect Server to various IBM Storage Protect Clients (including BA Client, HSM, VE, various types of TDP clients etc.) This automation utility is available in the form of a script that runs on the administrative client. Several schedules, with action=command are defined in this process of distribution and associated with nodes (clients) registered on the IBM Storage Protect server. These schedules are responsible for sending the certificate to the client, when the client scheduler runs those. The new certificate is then added into the client certificate database. Enabling the clients to trust the new certificate from IBM Storage Protect, for further communications, after the IBM Storage Protect switches over to the new certificate.

To automate the distribution of the IBM Storage Protect server certificate to clients, complete the following steps:

1. You must generate a new certificate on the IBM Storage Protect server for distributing certificate to all clients by using CREATE CERTIFICATE command.
   Log in to server administrative client (DSMADMC), and create the certificate by using the following command:

   CREate CERTificate "certificate_label"

   To know more details about CREATE CERTIFICATE command, refer to CREATE CERTIFICATE (Create a new TLS certificate).

   Important: Certificate file certificate_label.arm is created in the server's instance directory, where certificate_label is the label used while generating the certificate.
2. Copy the generated certificate file from server's instance directory to the admin client. For example, copy certificate_label.arm from server's instance directory to the admin client.
   Important: If server and admin client are on the same machine, you must provide the full path of the certificate file in the configuration file instead of copying the file path.
3. Place the script and configuration file in the same directory on the admin client machine.
4. Set appropriate values for different parameters in configuration file.
5. Ensure that the script has the execute permissions to the logged in user. You can also give permissions to the script for execution.
   For example, use the following command on Linux admin client :

   chmod +x cert_distribute.sh

6. Execute this script with dsmadmc user_id and password, along with the action argument as shown in the following examples:

   For Windows admin client :

   powershell .\cert_distribute.ps1 -id <dsmadmc-user-id>
                                    -pass <dsmadmc-user-password>
                                    -action <action-to-perform>

   For Linux /UNIX admin client:

   ./cert_distribute.sh -id <dsmadmc-user-id>
                        -pass <dsmadmc-user-password>
                        -action <action-to-perform>

   For AIX admin client:

   ./cert_distribute.ksh -id <dsmadmc-user-id>
                        -pass <dsmadmc-user-password>
                        -action <action-to-perform>

   Note: Please provide dsmadmc user ID and password with single quotes in case they contain special characters.

   For example:

   ./cert_distribute.sh -id 'test&44admin'
                        -pass 'Colt!44lifelate'
                        -action <action-to-perform>

7. To start a certificate distribution job, run the script with action distribute. Set of client schedules will be defined on server to distribute certificate to all clients.

   Example of executing the utility with distribute action on Linux admin client:

   # ./cert_distribute.sh -id admin -password passw0rd -action distribute
   
   ANS8002I Highest return code was 0.
   
   >> Schedules are defined to distribute the certificate to clients.
   >> A return code other than 0 indicates an error. For more information, take a look at the log file.
   >>The log file is located at /home/cert_distribute/cert_distribute.log
   >> Client nodes will begin receiving a new certificate after 5 hours from the start time of the scheduler window.
   >> To monitor the progress of certificate distribution, it is recommended to run this script in Report mode regularly for a couple of days.
   
   #

   Important:

   If an error occurs during the certificate distribution to client nodes, you must execute the cleanup script action to clear previous schedules before attempting the distribution again.

8. To monitor the progress of the certificate distribution job, run the script with action report. The distribute schedules will be run on a daily basis. You must leave the schedules to run for certain days if some clients are not reported as Completed.

   Example of executing the utility with report action on Linux Admin Client:

   # ./cert_distribute.sh -id admin -password passw0rd -action report
   
   The detailed status report is : /opt/tivoli/tsm/client/ba/bin/certdistribute.report.2023-11-02-09_49_19
   
   >> Any status other than 'Completed' indicates a failure.
   >> Client nodes with status ‘Missed’ specifies that the scheduled startup window is already passed.
   >> Client nodes with status ‘Future’ specifies that the beginning of the startup window is in the future.
   >> For ‘Missed’ and ‘Future’ status there is a chance that they will catch up as these schedules will be re-run on a daily basis.
   >> Please regularly monitor the progress of certificate distribution status, by executing this script in Report mode for a couple of days.
   >> Client nodes that are still in failed state, it is recommended to follow the manual steps for adding the certificate.
   >> 1. Copy the new certificate file over to the client box.
   >> 2. Execute below command to add the certificate.
   >> gsk8capicmd_64 -cert -add -label "<<New Certificate Label>>" -file "<<New Certificate File Path>>" -db dsmcert.kdb -stashed
   >> Tip: The dsmcert.kdb file is located in client installation directory. On Unix, Linux and Mac systems, if client sessions were ever started from a non-root user, copies of the certificate can be located in $HOME/IBM/StorageProtect/certs/ or in the directory determined by the PASSWORDDIR client option.
   
   #

9. After completing the certificate distribute job, run the script with action cleanup to remove the defined schedules from the IBM Storage Protect server.
   Note: You may not see some clients with Completed status after the retries as client scheduler might not run or there are some other issues. Such clients are considered failed. To distribute certification for the failed clients, refer to Renewing an SSL certificate of the IBM Storage Protect server .

   Example of executing the utility with cleanup action on Linux admin client:

   # ./cert_distribute.sh -id admin -password passw0rd -action cleanup
   
   ANS8002I Highest return code was 0.
   
   >> Cleanup Completed.
   >> A return code other than 0 indicates an error. For more information, take a look at the log file.
   >> The log file is located at /cert_distribute/cert_distribute.log
   
   #

10. You must confirm the distribution of certificate to all clients by executing script in report action.
    Note: The new and detailed status report is generated regularly. This report file contains the status of the schedule responsible for adding the certificate in client certificate keystore. For details, check possible values for status information at QUERY EVENT (Display client schedules)
11. Set the distributed certificate to default value after the report file shows the completed status for all clients.

    To get the new certificate in effect, the admin must execute the following command from the admin client:

    SET DEFAULTTLSCert <certificate_label>

    Where, certificate_label is the label mentioned in configurations file while executing the script. For details on SET DEFAULTTLSCert command, refer to SET DEFAULTTLSCERT (Mark a TLS certificate as the default)

12. Restart the server to get the new certificate in effect for further communication with all BA and TDP clients.
    Note: This results in aborting all the ongoing sessions (like backup or restore or replication) currently running on the server.