IBM AIX: Configure sendmail to use SSL/TLS with a self-signed certificate

Steps

1a. Inbound Connections

Follow these instructions to configure the sendmail server to support incoming requests using SSL/TLS.

Self-signed server certificate

Follow these instructions to configure sendmail to use a self-signed certificate.  All user input is highlighted in bold with an underline.  Substitute location information, the domain name and the sendmail server name as appropriate.

Notes:

• The server private key must not have a password.  This is specified by the “-nodes” option.
• The option “-days nnn” specifies the number of days for the certificate to be valid.  The default for the first “openssl req” command (which creates the authoritative certificate) is 30 days.  The default for the “openssl ca” command (which creates the server certificate) is 365 days.  The value in these instructions is 7300 days, or 20 years.

 
# cd /etc/mail
# mkdir CA CA/private CA/newcerts
# chmod 700 CA/private
# cd CA
# ln -s ../CA demoCA
# echo '01' > serial
# > index.txt
# openssl req -nodes -new -x509 -days 7300 -keyout private/cakey.pem -out cacert.pem
Generating a 1024 bit RSA private key
....................................................................................++++++
........................++++++
writing new private key to 'private/cakey.pem'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:Texas
Locality Name (eg, city) []:Austin
Organization Name (eg, company) [Internet Widgits Pty Ltd]:austin.ibm.com
Organizational Unit Name (eg, section) []:.
Common Name (eg, YOUR name) []:zlab123.austin.ibm.com
Email Address []:root@zlab123.austin.ibm.com
# openssl req -nodes -new -x509 -keyout private/sendmail_key.pem -out sendmail_certreq.pem
<Answer these prompts with the same responses as before.>
# openssl x509 -x509toreq -in sendmail_certreq.pem -signkey private/sendmail_key.pem -out sendmail_csr.pem
Getting request Private Key
Generating certificate request
# openssl ca -policy policy_anything -days 7300 -out sendmail_certificate.pem -infiles sendmail_csr.pem
Using configuration from /var/ssl/openssl.cnf
Check that the request matches the signature
Signature ok
Certificate Details:
        Serial Number: 1 (0x1)
        Validity
            Not Before: Aug 27 20:33:51 2018 GMT
            Not After : Aug 22 20:33:51 2038 GMT
        Subject:
            countryName = US
            stateOrProvinceName = Texas
            localityName = Austin
            organizationName = austin.ibm.com
            commonName = zlab123.austin.ibm.com
            emailAddress = root@zlab123.austin.ibm.com
        X509v3 extensions:
            X509v3 Basic Constraints:
                CA:FALSE
            Netscape Comment:
                OpenSSL Generated Certificate
            X509v3 Subject Key Identifier:
                DD:B3:C2:0E:F1:2F:24:F3:A3:FF:C4:D0:C8:51:92:49:9B:8E:9A:3E
            X509v3 Authority Key Identifier:
                keyid:F9:4A:4F:F8:BC:E8:7E:4C:B8:66:58:20:D4:B0:47:06:63:59:C5:F3
 
Certificate is to be certified until Aug 22 20:33:51 2038 GMT (7300 days)
Sign the certificate? [y/n]:y
 
1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated
# chmod 600 sendmail_certificate.pem private/sendmail_key.pem

Change the following entries in /etc/mail/sendmail.cf to match the path names used with the previous commands.  Note that “CACertPath” is a directory.

O CACertPath=/etc/mail/CA
O CACertFile=/etc/mail/CA/cacert.pem
O ServerCertFile=/etc/mail/CA/sendmail_certificate.pem
O ServerKeyFile=/etc/mail/CA/private/sendmail_key.pem

Vendor-supplied server certificate

If the server certificate is issued by a vendor or other certificate authority, they may give you instructions on how to create the appropriate server certificate request, for which they will generate a sendmail server certificate. They will also provide the CA certificate chain needed to verify the server certificate.  (Note:  if they request a server private key, it must not have a password because this will prevent sendmail from being able to use it.)

Once the CA certificate chain, server certificate and server private key files have been obtained, uncomment the following options in the /etc/mail/sendmail.cf file and set them to the appropriate full path names:

CACertPath:  the full path name of the directory containing the CA certificate chain file
CACertFile:  the full path name of the CA certificate chain file
ServerCertFile:  the full path name of the server certificate file
ServerKeyFile:  the full path name of the server private key file

O CACertPath=/directory/with/certificates
O CACertFile=/directory/with/certificates/cacert.pem
O ServerCertFile=/directory/with/certificates/sendmail_certificate.pem
O ServerKeyFile=/directory/with/certificates/sendmail_key.pem

1b. Outbound connections

For sendmail to support outbound SSL/TLS connections, obtain the CA certificate chain(s) needed to verify the server certificate(s) of the server(s) that the sendmail client will connect to.  Put all the certificates into a single file.  In this example, the file is assumed to be named /etc/mail/CA/cacert.pem.

Uncomment the following options in the /etc/mail/sendmail.cf file and set them to the appropriate full path names.  Note that the “CACertPath” option is for the directory which contains the CA certificate chain.

O CACertPath=/etc/mail/CA
O CACertFile=/etc/mail/CA/cacert.pem

2. Final Configuration

Stop and restart sendmail.

# stopsrc -s sendmail
# startsrc -s sendmail -a"-bd -q30m"

The following section is optional.

Verify that sendmail is using SSL/TLS by setting up syslog to capture mail.debug messages.  Once syslog is configured, edit the /etc/mail/sendmail.cf file to change the log level to 19.

O LogLevel=19

Stop and restart sendmail.  To test inbound SSL/TLS, send an e-mail from a different system which is also configured to use sendmail_ssl.  To test outbound SSL/TLS, send an e-mail from the configured system to another system that supports sendmail over SSL/TLS.  Messages like the following indicate success. If "verifymsg" is anything other than "ok", check the configuration settings and certificates.

Aug 27 15:43:36 zlab123 mail:info sendmail[3604654]: s9SIXanb3604654: --- 220 2.0.0 Ready to start TLS
Aug 27 15:43:36 zlab123 mail:info sendmail[3604654]: STARTTLS=server, get_verify: 0 get_peer: 0x0
Aug 27 15:43:36 zlab123 mail:info sendmail[3604654]: STARTTLS=server, relay=zlab122.austin.ibm.com [x.x.x.x], version=TLSv1/SSLv3, verify=NOT, cipher=DHE-RSA-AES256-SHA, bits=256/256
Aug 27 15:43:36 zlab123 mail:info sendmail[3604654]: STARTTLS=server, cert-subject=, cert-issuer=, verifymsg=ok

If any STARTTLS entries have messages, verify that all the configuration steps were performed exactly.

If there are no errors, lower the log level back down to 9 and restart sendmail.