A case study in scripted key management
A recent consulting engagement included a task to set up SSL channels between two queue managers. The standard operating procedure at this shop was to use interactive tools for nearly all administration tasks, and so my client had assumed that we would be using the iKeyman GUI for the creation of the keyrings and for all of the certificate tasks. They were a little reluctant when I suggested that we use scripts instead.
Their main objection was general unfamiliarity with scripting tools amongst the administration team. There was an underlying concern that a scripted solution would raise the minimum skill level needed to manage certificates. By comparison, the GUI was perceived as the easier and more expedient solution.
Beyond that, there was also a question of whether this was the best use of resources. I told my client up front that writing the scripts would take at least the same amount of time -- perhaps more -- as it would to walk their administrators through the same tasks using the GUI. The value of scripting is readily perceived in a large deployment, but with only two queue managers involved, the extra investment here was perceived as a waste of time.
Ultimately, it was business requirements that drove the decision. Since MD5
hashing has been broken, the client wisely stipulated that all keys must
be signed using SHA-1 instead. The GUI tools do not expose controls that
enable selection of the certificate signing algorithm, and they default to
Changing that behavior in the GUI requires editing the iKeyman and Java™ configuration
files. By comparison, the command line tools include the
which makes selection of the signature algorithm a trivial task.
When we finally got started, the client’s fears proved to be unfounded. The scripts are largely self-documenting and do not require sophisticated logic. They consist of nothing more than a commented list of commands with no branches or loops. Learning the scripts was just as easy for the administrators -- perhaps even easier than learning the GUI would have been. Also, our resource commitment was quite low since I already had some scripts from a previous assignment that were nearly perfect for what we needed.
We completed the SSL configuration in under three hours, and that included tailoring the scripts and training the staff on key management tasks. After about a month, the queue managers were migrated to new servers and rebuilt. The administrators were able to reuse the scripts we had created to set up SSL channels on the new queue managers in under an hour – unassisted -- using a process that is repeatable, consistent, and error-free. Our investment was recouped entirely with the first reuse.
In this installment of Mission:Messaging, I will walk you through the tasks to configure basic SSL channels with self-signed certificates, building two scripts as we go. Then, when it’s time to set up IBM® WebSphere® MQ SSL channels in your shop, you should be able to get your first channel pair up and running in under an hour as well.
First, a word about certificate types
The process to configure SSL with CA-signed certificates is similar to that for self-signed certificates, but the security policies created differ dramatically.
When using self-signed certificates, a policy of Deny-All is created. Each certificate added to the key ring is an exception to this policy. Granting access requires adding a certificate. Revoking access requires deleting a certificate. With self-signed certificates the MQ administrator must take specific action to grant access for a given certificate. All access control decisions rest with the administrator.
With CA-signed certificates, the effective policy is Allow-All for each trusted CA (Certificate Authority). In other words, adding a CA to a keyring establishes a trust policy for every certificate that CA has ever signed or will sign in the future. Narrowing down the scope of that trust requires use of the channel’s SSLPEER attribute to specify acceptable values for the Distinguished Name (DN) fields within the certificate. Revoking access requires a Certificate Revocation List, also known simply as a CRL. When a CA revokes a certificate, the action is published to a CRL. Customers of that CA mirror the CRL internally, where it can be checked by the queue manager when the channels start. If a channel start is attempted using a revoked certificate credential, the check of the local CRL copy will generate a hit and the connection will be refused.
With CA-signed certificates, the MQ administrator controls only the string matching template for Distinguished Names (DN) that are allowed to connect. Any new certificate issued by the CA with a matching DN value will be able to connect. The only control available to prevent this is the CRL. When using CA-signed certificates, maintaining CRL is considered mandatory because it is the only mechanism with which to exclude a certificate that has been compromised.
Many discussions of SSL characterize the choice of certificate types based on the cost or the process involved. Frequently, the approach is to prototype with self-signed certificates because they are free and easily generated, then switch to CA-signed certificates in later stages. The choice should really be made based on the desired security policy and whether the enterprise is prepared to set up and maintain a certificate revocation list. From a security perspective, it is generally better to start with a Deny-All policy and grant exceptions than to start with an Allow-All policy and build a blacklist. For these reasons, I prefer self-signed certificates and these are what will be discussed here.
For the purpose of this article, consider two queue managers, which I will refer to as QMA and QMB. At a high level, building a new WebSphere MQ SSL configuration with self-signed certificates requires these steps:
- Create the default keyring for QMA.
- Delete the default CA certificates from the keyring.
- Generate a self-signed certificate for QMA.
- Extract the public key of the new certificate.
- Repeat steps 1 through 4 for QMB.
- Import the public key of QMB into the keyring for QMA.
- Issue REFRESH SECURITY TYPE(SSL) on QMA.
- Repeat steps 6 and 7 for QMB.
Steps 1 through 4 are required for any new queue manager and are a natural choice for a single script. Although steps 6 and 7 could be scripted, there are times when it is necessary to import many public keys into a keyring, and not necessarily a good idea to issue a REFRESH SECURITY after each one. So, step 6 will be a script unto itself. We will document the requirement to issue a REFRESH SECURITY command, which can be executed from the command line or from your favorite administrative tool as you see fit. But this is largely a matter of personal preference, so feel free to add REFRESH SECURITY as the last step of the second script if you like. Once we get into the scripts, there are a few additional details, which I’ll cover as we go.
Common script headers
Although this article covers only a very small subset of key and certificate management tasks, the intent of scripting is to create reusable tools. For this reason, each script will contain a header in which you will set any variables necessary to execute the line commands.
Reusability also influences where you will be working with keyring files. For one-time use, it would be easier to build the keyrings in a local temporary directory, generate and exchange the certificates, and then move the finished keyring files to their final destinations. But you cannot generate a fresh keyring from scratch every time you want to add a certificate and, unless you have a secure deployment server and some really good automation, it is not practical to maintain a central repository of keyrings and push them out to your queue managers after performing maintenance. To be reusable, your script will need to manipulate the keyrings where they are actually used. This will require you to embed several file and path elements into your common script headers. Let’s take a look at those now.
Because it is customary for the queue manager name to be included in the certificate’s Common Name field, you will need a variable to hold it:
To complete the distinguished name of the certificate, a domain is usually required:
You will also need several organization and location fields. I have lumped these together in a single variable, called ORG:
The lower-case version of the queue manager name is required for the certificate label. This is what the queue manager looks for so it can locate the certificate it will present during an SSL negotiation:
You also need to know where the queue manager lives in the file system. WebSphere MQ will mangle certain characters in order to create platform-portable file names. For example, if a queue manager name contains a dot character, this is converted to an exclamation point. If your queue manager name contains any characters other than alpha-numeric, you will need to look in the file system to determine the mangled name and set that value here:
The script should be useable for non-standard installations, and in particular when the queue manager is implemented on a high-availability hardware cluster. The PATH2QMGR variable is used to hold this location.
The same variable on a Windows® system would have this value, assuming a default installation:
PATH2QMGR=C:\Program Files\IBM\WebSphere MQ
The password to the keyring is stored in the script as a user-editable environment variable. Be sure not to leave the script lying around with the password saved. A good place from which to save and run the script is in the queue manager’s SSL directory. This directory must be protected so that only the mqm (or platform equivalent) user ID can read the files. Since the key database password is stashed here, there is no additional exposure in storing the script here as well:
Password expiry is only used once but I like to create a variable for it and place the variable with the other user-edited fields. This calls out to the script operator the fact that a policy decision is captured here. The value is in days and I’ve specified a 10-year expiry for this example:
The command we will be using here is the GSKit C API command and not the Java implementation. This is because the C version exposes some options on the command line that are not available in the Java version. As an added benefit, the C executable is much faster because the Java version must build and tear down a JVM for each command invocation. The entire C-based script can run in the time it takes a single command to execute with the Java version. To make this work, the script just needs to know the path to the C binaries. For UNIX® systems, the default location is in /opt:
On Windows systems, the default is on the C: drive:
PATH=C:\Program Files\IBM\gsk7\bin:C:\Program Files\IBM\gsk7\lib:%PATH%
That was the last of the user-editable variables. Before running any commands though, you need to tell the script the name of the keyring file. That is built using a combination of the default WebSphere MQ settings and the variables entered in the previous steps. For UNIX and Windows, respectively, those variables are:
These locations are the default and should already be present in the queue manager’s SSLKEYR attribute. If alternate locations are chosen, be sure to update the queue manager accordingly.
The settings above can be used for any certificate management script. Some of the command invocations will require different or additional parameters, but the implementation stays the same: put the user-editable variables together up front where they can be managed as a set.
Create the keyring and remove the default CA entries
The next step is to generate the keyring in its default state. This requires a single gsk7capicmd command invocation:
gsk7capicmd -keydb -create -db "$MQSSLDB" -pw $PASSWORD -type cms -expire $EXPIRY -stash -fips
- The first gsk7capicmd parameter is the type of operation to be performed. In this case, a key database operation.
- The second parameter is always the action to be performed within the
operation type. Here, we intend to create a new key database which is
pointed to by the fully-qualified
- The database type is specified as
cmsbecause this is what the queue manager requires.
–stashoption creates a stash file with the password. This is how the queue manager is able to access the key database without human intervention.
- Finally, the
–fipsoption is specified. This not only generates a key database compliant with Federal Information Processing Security standards, but it also insures that the libraries and algorithms used when the command runs are FIPS compliant. I do not know of any negative impact to specifying use of FIPS-compliant commands and they do have the potential benefit of having been independently certified. Until I hear of a good reason not to use them in all cases, my policy is to always specify FIPS.
When the keyring is first created, it will be loaded up with a number of default CA signer certificates. Currently, these include root certificates from Entrust, Verisign, Thawte and RSA. As we saw earlier, the presence of these root CA certificates creates an Allow-All policy for every certificate that CA has ever issued or will ever issue in the future. Since we are using only self-signed certificates, all of these root certificates represent inappropriate access grants to our queue managers. Accordingly, the next step is to delete all of the default CA certificates.
The list of certificates changes over time. The specific certificates used here are from a V6.0 queue manager. Depending on your version and fix pack, the list you see could differ slightly. To insure that all of the CA certificates have been deleted, you will include a command to display all certificates in the keyring toward the end of the script.
The command to delete a certificate looks like this:
gsk7capicmd -cert -delete -label "Entrust.net Global Secure Server Certification Authority" -db "$MQSSLDB" -pw $PASSWORD -fips
–label option identifies the certificate by a human-readable string.
The label must be unique within a keyring and it is set when a certificate
is created or imported. The same certificate could have different labels
in different keyrings. In this case, the label identifies the specific
certificate to be deleted.
I’ve only listed one example here. The full script listing contains twenty-four additional commands that remove root and intermediate certificates of the various vendors.
Generate the local queue manager’s certificate
The next step is to generate the new certificate for the local queue
manager. The queue manager locates its certificate based on the label,
which must always be the fixed string
ibmwebspheremq followed by the
queue manager’s name folded to lower case. In the example below, the label
is derived by concatenating the fixed portion of the label with the
$FOLDEDNAME variable set in the script header:
gsk7capicmd -cert -create -db "$MQSSLDB" -pw $PASSWORD -label "ibmwebspheremq$FOLDEDNAME" -dn "CN=$QMGRNAME.$DOMAIN,$ORG" -fips
This command introduces another new option, the Distinguished Name of the certificate, which is abbreviated as DN. The DN is a globally unique name representing the entity represented by the certificate. It is composed of several sub-fields, including:
- CN=common name
- OU=organization unit
- ST=state, province
It is important to note here that the scope of "globally unique" is per issuer. It is perfectly valid for two or more certificate authorities to issue certificates with the same DN component. In the case of a self-signed certificate, each certificate is its own root and the only enforcement of global uniqueness will be the controls you build around the human processes for issuing and managing certificates.
The common name for queue managers is customarily the queue manager name, followed by the domain in which it resides. For example, QMA.us.ibm.com would be a valid CN for our QMA queue manager. The certificate might contain multiple Organization Unit (OU) entries. Careful selection of the OU naming helps to construct meaningful SSLPEER values. For example, if one of the OU entries is PROD or DEV, then it will be possible to construct an SSLPEER that prevents interconnection between production and development queue managers. (The channel’s SSLPEER attribute and Distinguished Names were discussed more fully in January’s Mission:Messaging column, Planning for SSL on the WebSphere MQ network.)
Extracting the new certificate
In order for the new certificate to be useful in an SSL negotiation, the remote queue manager must have access to the certificate’s public key. This is accomplished by extracting it to a file, copying the file to the remote node, and then importing it into the keyring residing there. Extracting the certificate will be the one of the last tasks in our current script, and then you will build a second script to import the certificate file into the remote keyring.
The command to extract the certificate is:
gsk7capicmd -cert -extract -db "$MQSSLDB" -pw $PASSWORD -label "ibmwebspheremq$FOLDEDNAME" -target $PATH2QMGR/$MANGLEDNAME/ssl/$FOLDEDNAME.crt -format ascii -fips
Here, you reuse the previous values for the keyring, certificate label, and
–target option identifies the location and name of the file
that will contain the extracted certificate. The .crt extension for the
file name matches the
–format ascii option and will enable you to import
the certificate later.
At this point, the work of the script is complete but the human operator needs to verify the results. You need to display the certificates in the keyring and ask the operator to check that no default CA certificates remain. This is required to catch changes that are made between releases of GSKit, which updates the inventory of default certificates:
gsk7capicmd -cert -list -db "$MQSSLDB" -pw $PASSWORD -fips
Second script: Adding certificates to the keyring
The final step is to exchange certificates between QMA and QMB. You will
build a second script that will read the .crt file created previously and
add it to the keyring of a queue manager. The preamble of the script will
reuse many of the variables you used earlier. The only difference will be
$FOLDEDNAME variable now refers to the remote queue manager and
not the local one. The command to add the certificate to the keyring
gsk7capicmd -cert -add -db "$MQSSLDB" -pw $PASSWORD -label "ibmwebspheremq$FOLDEDNAME" -file $FOLDEDNAME.crt -format ascii –fips
Notice that the script expects the file name to match the lowercase name of the queue manager and to be in the current directory. It also uses the same standard label for the certificate, even though this is not a requirement. The only time that WebSphere MQ requires the label to be in a specific format is for the queue manager to locate its own certificate. I chose to use the same label to represent a particular certificate across all keyrings because the consistency helps administrators recognize and manage the certificates easier.
As with the first script, the contents of the keyring are displayed to verify the operation has completed successfully.
The scripts provided with this article are examples of the minimum required to connect two queue managers. The design was such that the first script can be reused on any queue manager to create the keyring for the first time. The second script can be used repeatedly to add in certificates from any remote MQ node. The commands in these two scripts are sufficient to connect a network of any arbitrary size.
Of course, additional scripts can be easily built from these examples by substituting the appropriate commands. Some suggestions for you to try on your own include a script to delete a certificate, a standalone script to list the contents of a keyring, and a script to print the details of any certificate. Eventually, you might need a script to change the password of the key database, depending on the password expiry period used in your shop. Any of these can be created by making minor changes to one of the existing scripts.
The scripts I use are intentionally rather simple. Many of the fields in the scripts could be derived or parsed from the queue manager ini file or from the Windows registry with a little work. But as a consultant, I need tools that will run in the broadest range of environments possible, and which do not make too many assumptions. If you have the luxury of a permanent position, you might want to invest some time to make the scripts easier to use and more robust. Some enhancements you might consider adding would include logic to parse out things such as the location of the queue manager in the file system, or better error handling. The less the script user has to enter, the lower the probability of introducing human error. The better the diagnostics, the less chance of the operator missing an error message and continuing with a broken keyring. I’d be happy to host anything you come up with at t-rob.net, if you are inclined to share.
But whether you keep your scripts simple or enhance them, the automation will help you to implement SSL quickly, easily, and consistently.
|Code sample||0906_MQ_scriptingsample.zip||4 KB|
- GSKCapiCmd User’s Guide (PDF)
- Information Center: Working with WebSphere MQ TLS and SSL support
- Mission: Messaging: Planning for SSL on the WebSphere MQ network
- Gibson Research Corporation: High security password generator
- T-Rob.net, home of The Deep Queue, a podcast about WebSphere MQ security