Configuring SSL and TLS by using self-signed certificates

Configure Secure Sockets Layer (SSL) and Transport Layer Security (TLS) on the Tivoli® Storage Manager server, backup-archive client, and storage agent to ensure that data is encrypted during communication. You can use a self-signed SSL certificate to verify an SSL communication request between the server, client, and storage agent.

Before you begin

To use the SSL to secure communications between the Operations Center and the hub server, see Securing communications between Operations Center and the hub server.

Before you set up the server certificate on the client, follow these steps:

Open a command window and change the directory to your Tivoli Storage Manager client directory, for example: cd "C:\Program Files\Tivoli\TSM\baclient"
Append the GSKit binary path and library path to the PATH environment variable, for example:
```
set PATH=x:\Program Files\Common Files\Tivoli\tsm\api64\gsk8\bin\;
 x:\Program Files\Common Files\Tivoli\tsm\api64\gsk8\lib64;%PATH%
```
where x: is the system drive where Tivoli Storage Manager is installed.

Tip: If your client operating system is 32-bit, replace the gsk8capicmd_64 command with gsk8capicmd in all GSKit commands.

About this task

You can restrict SSL communication to use TLS 1.2 and prevent the use of previous TLS protocol levels, which are less secure. For successful TLS communication, configure the client that communicates with the target server or storage agent to use TLS 1.2.

Tip: Enter commands on one line. In the following steps, commands are displayed on multiple lines to make it easier to read. Ensure that you enter a space after each command.

The image is a graphical depiction of how you configure SSL by using self-signed certificates, and provides the number for each task step.

Procedure

Specify the TCP/IP port on which the server waits for client communications that are enabled for SSL or TLS.
- Server: Update the dsmserv.opt file in the server instance directory by specifying the SSLTCPADMINPORT and SSLTCPPORT options, or both. To use TLS 1.2, specify the SSLTLS12 YES server option in the server options file.
- Client: Specify the ssl yes and tcpport options in the client options file:
  - dsm.sys
  - dsm.opt
  The server is typically set up for SSL or TLS connections on a different port. If you use an SSL or TLS connection, two ports are open on the server. One port accepts regular non-SSL or non-TLS client connections and the other port only accepts SSL or TLS connections.
- Storage agent: In the dsmsta.opt file, specify the SSLTCPADMINPORT and SSTCPPORT option, or both.
Create the key database file and establish SSL communication between the server and the storage agent if necessary:
- Server: Start the server. This action creates the server key database file, cert.kdb, and stores it in the server instance directory.
  If a password exists for the server database, it is reused for the cert.kdb key database. After you create the database, the key database access password is generated and stored. To establish communication between the server and the storage agent, issue the following command:
```
define server sta
hla=ip_address
lla=ssl_port
serverpa=password
ssl=yes
```
- Client: Create the key database file, dsmcert.kdb, by issuing the following command. Issue the command from the bin directory:
```
gsk8capicmd_64 -keydb 
-create -populate
-db dsmcert.kdb 
-pw password -stash
```
  Tip: The bin directory for the client is installed to the client system directory during client installation. For example, the bin directory for the client is installed in the following path:
```
system directory\Tivoli\TSM\api64\gsk8\bin
```
- Storage agent: Initialize the storage agent and add communication information to the device configuration file and the storage agent options file dsmsta.opt by issuing the DSMSTA SETSTORAGESERVER command. You must specify the SSL=YES and STAKEYDBPW=password parameters to create the key database file in dsmsta.opt. All passwords are encrypted in dsmsta.opt.
```
LDR_CNTRL=TEXTPSIZE=
64K@DATAPSIZE=64K@STACKPSIZE=64K@SHMPSIZE=64K
dsmsta setstorageserver myname=storage_agent_name
mypa=sta_password
myhla=ip_address
servername=server_name
serverpa=server_password
hla=ip_address
lla=ssl_port
STAKEYDBPW=password
ssl=yes
```
```
dsmsta setstorageserver 
myname=storage_agent_name
mypa=sta_password
myhla=ip_address
servername=server_name
serverpa=server_password
hla=ip_address
lla=ssl_port
STAKEYDBPW=password
ssl=yes
```
  Tip: Specify the SSLTCPADMIN port for the LLA parameter. If you do not specify the SSLTCPADMIN port, use the SSLTCPPORT port instead.
Create the key database certificate and default certificates by starting the storage agent.
Stop the server and the storage agent.
Enable SSL or TLS communication by importing the .arm files. You must use the same communication protocol between the server, client, and storage agent.
- Server and storage agent: Import the storage agent certificate to the key database file for the server and import the server certificate to the key database file for the storage agent:
```
gsk8capicmd_64 -cert 
-add -label 
server_example_name
-db cert.kdb -stashed 
-file cert256.arm 
-format ascii
```
  Tip: To use TLS 1.2, take the following actions:
  - Ensure that the default label is "TSM Server SelfSigned SHA Key."
  - In the dsmserv.opt and dsmsta.opt files, specify the SSLTLS12 YES server option.
- Client: Import the cert.arm file and the cert256.arm file to the key database file:
```
gsk8capicmd_64 -cert -add -db dsmcert.kdb 
-stashed -label "TSM server server_name 
self-signed key" -file path_to_cert256.arm 
-format ascii
```
  To change the default certificate in the cert.kdb key ring database file to the "TSM Server SelfSigned SHA Key" label, issue the following command from the server instance directory:
```
gsk8capicmd_64 -cert 
-setdefault -db cert.kdb 
-stashed -label 
"TSM Server SelfSigned 
SHA Key"
```
  Tip: To use TLS 1.2 communication between the client and the storage agent, the default label must be "TSM Server SelfSigned SHA Key."
Restart the storage agent and the server.