It seems every day we are bombarded with stories in the technical and mainstream press of attacks of computer systems where passwords are stolen, after which these passwords are then available to attackers. The press frequently points out that one reason that these password data was retrievable is because “they were not encrypted.” In this situation, it’s important to recognize that the passwords in question are user passwords, those used to log in and access the system, not passwords associated with the system processes and binaries. This is an extremely important difference. As a result I will argue here that the premise “unencrypted passwords are bad” — without additional context — is technically inaccurate and misleading.
With such press coverage in mind, perhaps the most common comment received by WebSphere security consultants, by our technical sales people, and by the WebSphere security development architects, centers around customers wanting to encrypt passwords in their IBM WebSphere Application Server systems. Repeatedly, the message that we get is, “WebSphere is mostly meeting its claim of being secure by default, but there are passwords stored on the filesystem which are simply encoded, not encrypted.” This is frequently followed by “and we failed our security audit because they are not encrypted.”
First, it is the position of IBM Software Services for WebSphere and IBM WebSphere Development that, from a security perspective, the encoding of system passwords is not an exposure. Auditors who make these claims do not understand the security implications of what they are asserting to be a preferred solution. I will justify this position shortly.
However, since some clients have insisted that encryption of system passwords used by WebSphere Application Server is preferred over encoding, a System Programming Interface (SPI) has been available in WebSphere Application Server since Version 6.0.2 which can be leveraged to implement whatever password “hiding” solution a client might want to implement. Using this SPI, IBM Software Services for WebSphere has implemented such a solution for a number of clients. I will first define some basic security concepts, and then describe the design considerations in this custom solution.
The basics: Encoding versus encryption
What is the difference between encoding and encrypting? We use the term
encoding to apply a scheme where something is hidden by formula or algorithm.
Consider an example of a secret password that you want to “hide.” For
illustration, suppose I have an admittedly very bad password:
SecretMonkey. A reversible algorithm could be used to encode this phrase to “hide” its true value.
Julius Caesar is attributed to have used an encoding scheme called a Rotation cipher or a ROT cipher, where the numerical position in the alphabet is shifted by some number. History now calls this a Caesar cipher. The ROT3 cipher maps 3 character positions to the right. The mapping for ROT3 (for an English alphabet) is:
My ROT3 ciphered password above then becomes:
When presented with this encoded string, professional cryptanalysts would quickly guess the encoding algorithm and decode it with the reversing algorithm (ROT-3). The strength of encoding is in the secrecy of the algorithm. It is important to remember that attackers are adept at reverse-engineering algorithms. Once that algorithm is known the data is unprotected.
Encryption, as opposed to encoding, bases its security not on the algorithm used, but on the strength of hard-to-guess keys. There are still algorithms used, but these algorithms are (typically) widely known and understood. Encryption algorithms differ from encoding algorithms in that they take two inputs – the text and a secret key. In much the same way that a 4 digit debit card PIN uses a 1 in 10,000 possible (104) choice secret, encryption keys use a secret key which takes a lot of effort to brute-force guess.
There are two types of encryption, and both are based on complex mathematics. The first type of encryption involves a single key –called Shared Key, Secret Key, or Symmetric Key Encryption. Consider two functions:
cipherText = Encrypt(plainText, secretKey);
and the reverse:
plainText = Decrypt(cipherText, secretKey);
To protect my data, I chose a key from a very large possible set of keys (for example, a 256 bit key, which is a set of 2256 possible numbers, which is a huge number; in decimal notation that is 1.15x1077: a 115 followed by 75 additional digits). As you can imagine, it would take someone a long time to call the Decrypt function iterating though all possible keys.
The second type of encryption uses two different, or asymmetric, keys. This type of encryption is known as Public Key or Asymmetric Encryption. These two keys are related to each other in a complex mathematical relationship. Without going into the mathematics, the function call to encrypt some text is:
cipherText = Encrypt(plainText, KeyA);
and the asymmetric reverse is:
plainText = Decrypt(cipherText, KeyB);
The important point here is that the strength of both types of encryption rest on the difficulty of guessing the key argument needed for that decrypt function – either the shared “secretKey” for data encrypted using Symmetric encryption, or the “KeyB” for Asymmetric Encryption.
I want to provide a little more “theory” here and introduce the idea of something called a one-way hash function. Again, I will not go into the mathematics, because the details are not actually important to my argument. A one-way hash function (again) uses a well known, understood, algorithm:
digest = hash(inputString);
where the inputString is transformed into an output called a digest, and where it is computationally infeasible to reverse the process (hence the term "one-way"). Every time you provide the same inputString, you get exactly the same digest. The digest usually is a constant length for a particular hash function. Hash functions have the additional property that if you change any one bit in the input, you get a completely different digest output.
A trivial hash function could produce an 8 bit digest. Every input “string” will produce one of 28 digests. If one bit of the inputs changes, a different digest is produced. If you are still paying attention, you will realize that there are an infinite number of input strings that produce the same digest. But we know that from a given digest it is impossible to guess the inputString. When the size of the digest becomes big enough (for example, 2256 bits), the chances of another random input producing the same digest are infinitely small.
Attackers realized that for a known algorithm, it is possible to pre-generate all possible input, and calculate their own digests. These were known as “rainbow” tables. In practice, implementers change the hash function behavior by concatenating any input string with a secret/different prefix called a salt. So the preferred usage is:
digest = hash(inputString||salt);
An attacker would have to build rainbow tables with all possible salt strings.
Why, you might be asking, am I meandering, and not getting to my point? Those press reports of stolen password databases are talking about systems where the users’ passwords are checked against a database, and sadly, that data in the database was not properly protected. It is well accepted knowledge in the IT field that those passwords should not be stored in plaintext, but that they should contain the digest of the password with salt. In other words, a password “repository” should not contain my password string “SecretMonkey.” When validating my password, the system should concatenate the particular salt that it uses to my string, compute the hash of that concatenated string, and compare it against the saved password digest. If those two digests match, there is an extremely high probability the inputString – “SecretMonkey” – was my password.
A fictitious example of hashing my string, with different salts might be:
- SecretMonkey (without salt) > !GHD&@DB&!DJD
- SecretMonkey||(salt=1111) > UCHAKJ$HDJS
- SecretMonkey||(salt=2222) > njurnasiurrjfdd
It is crucial that your security audit ensures that these passwords are stored in hashed form. Notice I did not say encrypted. I hope you understand the subtle distinction (one is reversible, the other is not).
But I am not finished. Without question, I support anyone who says that end users passwords must be stored as salted, hashed digests. But there is a colossal difference between end user passwords and system passwords.
User passwords are not system passwords
When a WebSphere system “fails” an audit because the system passwords are not “secure” the auditor is not seeing “user” passwords. These are the passwords associated with the operating system user ID that WebSphere Application Server runs as or the passwords that WebSphere Application Server applications use to connect to other systems. For example, when I login to the “Financial” application, I can imagine that following happens:
- I am authenticated against the LDAP. My userid and password are used in an LDAP bind operation. That LDAP server hashes the password (along with its unique salt) and compares the resulting digest against my saved digest to confirm or reject my claim of identity.
- Once authenticated, the web application connects to some backend databases to find new “notifications” and “network requests.” This could be an SQL command similar to “select notifications where userid=Lansche”. To prevent any computer within the network from having access to this data, the “Financial” application connects to that database with “FinancialApp” credentials. For the sake of argument, let us assume that these credentials are a userid and password.
Applications servers need passwords to perform LDAP searches, to connect to databases, queue managers, web services, and other applications, and for applications’ runAs identities. When the application server authenticates to these other systems with a userid and password, the password needs to be sent in a “usable” format — the plain-text version of the password. As my colleague T. Rob Wyatt explains in his blog:
|From: Store and Forward: Encrypting passwords in config files – secure or not?|
Ultimately, it comes down to this: any system that must boot to operational status unattended must have access to a useable authentication credential …“Unattended” means that the system must go from powered down to fully operational without human intervention. Is it acceptable that after rebooting a server all the applications start up a command prompt waiting for a human user to enter all the passwords? Usually not. Overwhelmingly not. There are some cases where this is true but I don’t have the security clearances to work with those systems and if I did I couldn’t write about it anyway. For everyone else we all need systems that recover automatically and without human intervention. Bottom line, “unattended” means that the credentials must be stored where the application can access them at run time, usually in the file system.
In other words, when the “Financial” application starts up you have two choices:
- Wait for an operator to type in at a console the password for the “FinancialApp” userid (and every other system account needed), or
- The application obtains the password from somewhere on the filesystem.
I am confident that real applications follow the second model. Most non-military/national-security systems follow the second model.
PCI-DSS and system passwords
So, your auditor says that those system passwords must be encrypted. The PCI-DSS does not include that particular requirement. If fact, the only thing they say about system passwords is, “Do not use vendor-supplied defaults for system passwords and other security parameters.”
Figure 1 Table from PCI DSS - Requirements and Security Assessment Procedures, Version 2.0
There are certainly other mentions of “passwords” in the specification – but those relate explicitly to user passwords. For example, requirement 8 mandates that you “Assign a unique ID to each person with computer access,” and as part of that, section 8.4 states that passwords must be “unreadable during transmission and storage on all system components using strong cryptography.” This is critical – user passwords should not be stored anywhere unhashed; indeed as I mentioned above, the only storage should be a salted one-way hash! However, this refers explicitly to user passwords, not to system passwords.
Why encrypted passwords are no more secure than encoded passwords
But suppose your auditor insists. Let us analyze the process the system now goes through logically.
- Assume that encoding passwords is not sufficient. Why? Because anyone with read access to the filesystem has access to the credentials:
- Perhaps users have access to the filesystem, or
- Applications running in WebSphere Application Server can read the passwords.
- You implement the password encoding SPI, and encrypt the password. To do this, you need a key. The password is now stored encrypted on the filesystem. Assume a symmetric encryption (it is faster than asymmetric encryption). You store the key in a keystore.
- To be able to convert from the encrypted text back to plain text the server needs the decryption key. The server reads the key from the keystore. To prevent any application from being able to trivially get that key from the keystore, you password protect the keystore. WebSphere Application Server needs access to that password to be able to read the keystore. Where is that password saved? Because prompting an operator is not acceptable, that password is stored (drum roll please) in a file on the file system; a file containing a password to enable an application to read a keystore is known as a “stash” file.
But wait! We said in step 1, that our fundamental issue is that we are unable to control what processes have access to the filesystem. We still have that problem. Now it is slightly more inconvenient, but all the pieces necessary to get this system password remain on the filesystem and accessible to WebSphere Application Server or applications running inside the application server:
- The password as cipher text within the WebSphere Application Server configuration files.
- The keystore containing the key needed to decrypt that cipher text.
- The stash file containing the password to read the key from the keystore.
We have only moved the problem two steps to the right. This is no more secure than the passwords simply being encoded in the file. In fact, T-Rob, in his blog entry makes a compelling argument that encrypting system passwords lowers the system’s security.
The real security issue to be addressed is controlling who (people or applications) has access to the filesystem. The best way to address this real security issue is discussed in my hardening paper. (Hint: use file system access control.)
But, if you (still) insist...
The interface for defining your own password handler is the com.ibm.wsspi.security.crypto.CustomPasswordEncryption interface. You create a com.ibm.wsspi.security.crypto.CustomPasswordEncryptionImpl class, and place the class in the <WAS>/lib/ext directory. WebSphere Application Server processes will use this class without further administrative configuration. There can be only a single class that implements the interface. This interface defines the following methods (as expected):
EncryptedInfoencrypt(byte decrypted_bytes) throws PasswordEncryptException byte decrypt(EncryptedInfo info) throws PasswordDecryptException void initialize(java.util.HashMap initialization_data)
This last method is reserved for future use and is currently not called by the WebSphere Application Server runtime.
WebSphere Application Server interacts with these passwords in two ways: it stores the files when the passwords change, and it reads the files to obtain the password.
When WebSphere Application Server stores a password that has been updated, it checks to see if a custom encryption provider is available. If a provider is available, it will be given the cleartext password. The provider should manipulate the password in some way and return to WebSphere Application Server binary information that is meaningful to the provider. WebSphere Application Server then stores this binary information in an appropriate XML file as usual. The stored data indicates that it was encoded using a custom provider. This impacts future reads.
When WebSphere Application Server needs a password, it reads the configuration information (stored as XML files) and examines the information to determine if a custom provider was used. If the password is encoded in the default manner, WebSphere Application Server decodes it internally, as usual. If a custom provider has been used, WebSphere Application Server will pass the encoded binary information to the user-defined provider if one is available (if one is not available an error will occur). It is then the responsibility of the provider to return to WebSphere Application Server a clear text password that is represented by that binary data.
This model of lazy read/write means that at any given time in a WebSphere Application Server cell there might be passwords encoded using the default mechanism and other passwords encoded/encrypted using a custom mechanism. And, should the provider allow for updating the encryption key (as ours does), it is possible that multiple passwords might be stored within the same cell that are encrypted using different keys. Thus, the provider must be able to decrypt a password encrypted using any previous encryption key. Our provider addresses this by keeping multiple key versions and by embedding the key version in the information about the encrypted password.
Our provider design highlights
The remainder of this article documents the IBM Software Services for WebSphere custom password encryption provider. This is not a freely available piece of code, and there is no attachment that you can download with this article. Rather, this is an IBM Software Services for WebSphere services asset and is available by engaging with IBM Software Services for WebSphere in a billable services engagement. During this engagement, an IBM Software Services for WebSphere consultant will work with you to understand and document your requirements and tailor the asset if needed to better meet your use cases. They will then leave you with your own copy of the asset code, which you can then use in your enterprise.
The key driver in making this code available as an IBM Software Services for WebSphere asset is that, while there is very little actual security value in using this code, the true value is in enabling clients to pass a security audit where this issue has been raised, despite the fact that the encoding of system passwords is not a security exposure. In our experience, if the IBM Software Services for WebSphere security consultant is allowed to interact with the PCI auditor during the course of the engagement, there is a good chance that he or she will be able to educate the auditor on the difference between system and user passwords and negate the requirement to deploy (and therefore own and be responsible for) the code!
The key design decision in the IBM Software Services for WebSphere custom password encryption provider implementation was to leverage the existing Java JCE infrastructure as much as possible. Thus, the symmetric keys that are used for password encryption and decryption are stored using the Java KeyStore support. We specifically chose to use the JCEKS format, as this format properly supports symmetric keys. Java KeyStores allow for the keys stored in the keystore to be encrypted using a password, and they also allow for the keystore file to also be encrypted using a password. Rather than asking a human being to come up with an appropriate password, we automatically generate a very long (30+ characters) alphanumeric password and stash that in a second file – our stash, or master password file. To prevent trivial snooping, that file's contents are XOR'ed (using a four character XOR mask) to make guessing difficult.
When WebSphere Application Server needs to encrypt any system or WebSphere Application Serer password the provider is called. The provider is expected to return a string and encrypted bytes. (It is important to note that if you are using the file-based repository in the Federated Repositories user registry, user passwords are one-way salted hashed passwords. WebSphere Application Server does not use this SPI for writing those user passwords). When WebSphere Application Server needs to decrypt a password, it checks to see if the custom provider or default provider was used. If the custom provider was used, it is given a string (defined by the provider) and bytes. The provider’s job is to return the password as a char array. We take advantage of that string to store meaningful information.
Be aware that WebSphere Application Server does not automatically encrypt all passwords. It will only start using the code for new encryptions, so our code added functionality to force WebSphere Application Server to re-encrypt all passwords.
In order to enable managing and periodically updating the encryption keys without having to stop the entire cell, we need to support multiple encryption key versions at the same time. To do this, we embed in the string given to WebSphere Application Server the key version used to encrypt that password. A class we called the EncryptionKeyManager maintains a set of encryption keys (ordered by creation time). Whenever a string is decrypted, the key used to encrypt is used (assuming it is still available). When a string is encrypted, the latest key is used. When servers are started, they read into memory the latest version of the keystore and, thus, all of the current keys. In the event that a decryption fails because of a missing key version, the CustomPasswordEncryptImpl class will automatically reload the keystore to see if there is a newer key available. This might happen if a new encryption key is added (and the DMgr is restarted) but the application servers are not restarted. Thus, our solution does not require application server restart after the initial configuration to effect changes to the keys used for encryption.
Strong cryptography is obviously a key requirement of this solution. Therefore, we made the following decisions regarding cryptography usage:
- We use the Java JCEKS keystore provider, as it is capable of storing symmetric keys.
- We use a Java JCE Cipher that uses AES encryption with Cipher Feedback Mode. We also use no padding to avoid adding characters to passwords. The precise provider string is therefore AES/CFB/NoPadding (see the JCE documentation).
- Since we are using a CFB algorithm, we must also store the initialization vector used for encrypting data; we allow the JCE runtime to create a random initialization vector each time a Cipher is used to encrypt a password. We then store that vector with the password for initializing the Cipher before decryption. Using a separate vector for each password limits its reuse and improves the security of the encryption. Even if the same password is encrypted using the same key, the resulting encrypted data is dramatically different.
- We use 128 bit key lengths for the AES encryption. Longer key lengths require special JDK JCE jurisdiction files. These special files are limited by US Federal export laws. Given the amount of data being encrypted and the inherent weakness in the approach (storing the decryption keys on the same file system), we do not believe longer keys provide any significant improvement in security.
- We add random salt to the start of each password prior to encryption to further randomize the information stored, making decryption even more difficult.
We have chosen not to make the keystore, algorithm, or encryption type configurable, as doing so would significantly complicate the implementation, and also raise the possibility that someone could accidentally configure very weak password storage. Since all configuration information is managed by a single class, anyone with source code access (and appropriate cryptographic skills) can change the code to use other algorithms or configurations.
One obvious question to consider when adding support for the encryption of passwords is what impact this might have on performance. Cryptography is, of course, expensive and performing cryptographic operations could have a serious impact on the performance of applications. Fortunately, WebSphere Application Server aggressively caches (in memory) decoded/decrypted passwords. Therefore, while there will be a slight performance overhead when starting servers (while they read the passwords) and when making administrative changes that impact passwords, runtime performance should not be effected by this solution.
A final concern pertains to the keeping of passwords in memory. Unfortunately, while the concern is valid, there is simply no feasible way around this issue. As required by the J2C specification, every request to a resource must first check with a login module to obtain credentials for the back end resource. The login module must provide, among other things, the cleartext password so that the underlying resource can use it (this only applies to resources that need passwords). If passwords were not remembered in memory decrypted, every call to getConnection() on every resource from every application would result in a password decryption. That would likely have a significant performance impact. Since getting passwords from a memory image of a process is not easy, we consider this risk acceptable given the very real performance benefit.
As argued above, it is important to remember that the approach described in this article does not significantly increase the security of a WebSphere Application Server environment. At best it slightly increases the difficulty of gaining passwords. Prior to the use of this enhancement an attacker had to copy a password-containing file and then figure out how to undo the XOR and base64 encoding. With the use of this feature, an attacker has to copy the password-containing file and the encryption key files (which must be on the machine and must have the same read access permissions as the password file). Then, the attacker needs to use these files together to decrypt the passwords. While not trivial, it is not necessarily difficult for someone with the right skills. The real value of this solution is two fold:
- First, it creates the illusion of stronger security which is sometimes necessary to meet corporate security guidelines. Of course, the passwords are just as secure when simply encoded as when they are encrypted.
- Second, in the cases where the WebSphere Application Server XML configuration files are shared with third parties, storing the passwords in an encrypted form does prevent those third parties from determining the real value of passwords embedded in the XML configuration files. This presumes, of course, the third party is not provided a copy of the encryption keys — this is why this solution stores the encryption keys outside of the XML configuration tree.
When backing up the system, some consideration should be given to the fact that the backups contain both the encrypted passwords and the decryption keys. For maximum security, you should carefully protect those backups and even consider storing the decryption keys on a separate backup. If you choose the latter option, be very careful not to lose those keys. The passwords cannot be recovered without the decryption keys!
The decision to encrypt passwords using secret keys leaves open the possibility of a total and unrecoverable cell outage in the event of the loss of an encryption key. Therefore, it is crucial that you backup your cell prior to any key management operations and take great care when performing any key management activity. If an action results in the loss of encryption information that is still in use, there is no way to recover without backups.
Managing the keys: the passmgr utility
A simple command line utility (passmgr.bat/sh) is provided to manage the encryption data. The launch script has UNIX® and Windows® versions. The passmgr script follows the standard documented "thin client" launch approach.
After copying passmgr.bat/sh to some directory and editing the script to specify the correct value for WASHOME, the tool is launched quite simply by using passmgr.bat/sh, for example:
In most cases, you will want to provide the profile name of the WebSphere Application Server profile the tool should work with. If no profile is specified, the default profile will be used. To specify a profile, start the passmgr.bat with a profileName. For example:
passmgr.bat -profileName Dmgr01
After specifying the profile name, there are a number of additional optional arguments, as well as the mandatory command. The commands available are listed below; if you try to run the tool without providing a command, you will get an error message indicating your options:
passmgr.bat No arguments specified Usage: <command> [-dir <dir>] [-key <key#>] [-host <dmgr host>] [-port <dmgr port>] [-trace] where <command> is one of: create - create keyfile addkey - add a key to keyfile info - print information about a keyfile deletekey - delete a key from keyfile scanall - scan/read all passwords (may aid in debugging app server issues forceencrypt - force the reencryption of all existing passwords changemasterkey - change the master key protecting the keyfile
The optional arguments have the following meanings:
- -dir: Specify an alternative root for the keyfiles. This is useful if you want to create a keyfile, but do not want to update the keyfiles used by the current profile. It is handy for testing.
- -key: For commands that require a key number (for example, deletekey) this is the key number.
- -host: For those commands that talk to the dmgr (for example, scanall and forceencrypt), you can specify a non-default dmgr hostname. The default is localhost.
- -port: For those commands that talk to the dmgr (for example, scanall and forceencrypt), you can specify a non-default dmgr port. The default is 8879.
- -trace: Output trace information to the console to indicate what the management tool is doing.
IBM WebSphere Application Server (and products built on top of WebSphere Application Server) stores system account passwords in a variety of files within the WebSphere Application Server profiles file structure. These passwords are encoded, not encrypted. Encrypting these passwords does not increase security, nor is such encryption required by the PCI-DSS standard or other standards, but IBM is frequently asked to support password encryption.
WebSphere Application Server provides an SPI for customizing the processing of these system passwords. This article described an IBM Software Services for WebSphere custom implementation of this SPI.
If you are interested in deploying this custom implementation from IBM Software Services for WebSphere, or in learning more about WebSphere Application Server security, contact IBM Software Services for WebSphere to discuss the possibility of engaging with a security consultant, or for a customized on-site class in WebSphere Application Server security. The class covers security hardening, customizing authentication, integration, single sign on, and a variety of other related topics in depth.
I would like to thank my colleagues Bill O’Donnell, Tom Alcott, Simon Kapadia, T. Rob Wyatt, Pete Neergaard, David Mundhenck and Paul Glezen for their valuable input and assistance.
I would especially like to thank Keys Botzum who wrote the original asset, and original drafts of a significant portion of the text above.
- WebSphere Application Server security
- Advanced Authentication in WebSphere Application Server
- Advanced security hardening in WebSphere Application Server V7, V8 and V8.5, Part 1: Overview and approach to security hardening
- Information Center: Plug point for custom password encryption
- Sun JCE documentation
- Wikipedia: Block cipher modes of operation (cryptography)
- Payment Card Industry (PCI) Data Security Standard – Requirements and Security Assessment Procedures – Version 2.0 Oct 2010
- Encrypting passwords in config files – secure or not?
- IBM developerWorks WebSphere