Not long ago a colleague told me he wished that he could use a .kdb format keystore for his Java applications. When I inquired as to why, he said he liked that the .kdb includes the ability to stash an encrypted version of the password, whereas with his Java application he was obliged to store the password in a configuration file and, more importantly to his mind, in plain text. My initial reaction was that encrypting the Java passwords would probably be a good thing. Judging by the frequency with which this requirement comes up, I’m guessing most people would agree. Intuitively, it makes sense – an encrypted password must be more secure than one in plain text, right? The more I think about it, the more I’m convinced that the opposite is the case. I’ll explain why after the break.
Before we get started, let me put some scope on this. I am NOT talking about password databases here. Any time that many passwords are stored, they should always be encrypted using individual salt values and iterated hundreds of times with an irreversible hash function. OWASP has a lot to say on that topic and I won’t attempt to cover it here. What I am talking about are passwords used to authenticate an application to something else. Typically in the MQ world these are passwords to a Java keystore and the application needs at most one or two of them. The question then is how much benefit is derived from encrypting a single password in a configuration file? Arguably in many cases that achieved benefit is actually a net negative due to weaknesses in password management.
Ultimately, it comes down to this: any system that must boot to operational status unattended must have access to a useable authentication credential.
Let’s examine that in more detail.
“Unattended” means that the system must go from powered down to fully operational without human intervention. Is it acceptable that after rebooting a server all the applications start up a command prompt waiting for a human user to enter all the passwords? Usually not. Overwhelmingly not. There are some cases where this is true but I don’t have the security clearances to work with those systems and if I did I couldn’t write about it anyway. For everyone else we all need systems that recover automatically and without human intervention. Bottom line, “unattended” means that the credentials must be stored where the application can access them at run time, usually in the file system.
“Useable” means that regardless of how and where it is stored, the credential must be capable of authenticating and/or encrypting and decrypting to and from adjacent systems or applications. In order to do this the application presenting the credential must be able to render an encrypted credential back to its plain text form. If the credential can be rendered to plain text using only a defined procedure we say that it is obfuscated. This is not that much better than a plaintext password since anyone who can read the obfuscated value can derive the plain text password. If the credential has been encrypted then you need both a defined procedure and a key to decrypt it. Arguably this is better because the attacker needs both the decryption process and the password decryption key. But where do you store the password decryption key? Compile it into the program? Store it in the filesystem? You could of course encrypt it, but then you’d be left with yet another key and no place to put it. Bottom line, “useable” means that no matter how many layers of protection you put on a stored credential, the application must be able to unwind them and render the credential in plain text.
The combination of these requirements for unattended access to usable credentials ultimately means that if an attacker can read the filesystem, compromise the running operating system, or compromise the running application, then the credentials are compromised no matter how many times they’ve been obfuscated or encrypted.
Sure, there are ways to mitigate this risk. For example, you might store the application credentials remotely, for example on a network hardware security module. But you don’t access that Network HSM over plaintext do you? No, you do so using TLS and authenticate with some kind of credential…which is stored on the filesystem. Perhaps you encrypt the filesystem underneath the application? OK but then to access that encrypted volume requires a credential…which is stored on the filesystem. No matter how much obfuscation or encryption is applied to a password, if the application can render it, so too can an attacker.
If I’ve convinced you of anything so far, it is that encrypting individual application passwords doesn’t add to their effective security. True, it ups the attacker’s skill requirement a bit but that just makes it less convenient for legitimate users. A deliberate attack by a skilled adversary won’t be slowed down much, if at all. But does encryption actually make the system less secure? Perhaps.
To understand why, consider the administration required when passwords are encrypted and let’s use keystore management as our example case. Legitimate administrators must know the passwords to keystores in order to perform routine maintenance. If the local password is opaque to the administrator then there must be some means of accessing the plaintext version of it, for example when it is time to renew a certificate. One approach to this is to use a single password across all systems and that is well-known to all administrators. This tends to lead to short, human-readable passwords that are subject to brute-force attacks, and of course compromise of one node leads to compromise of the entire network. As bad as this sounds, you might be surprised to know how many shops use something like “mqm2011” as their password.
So what is the alternative? Some shops use secure password storage. This too is usually protected by a human-readable password and again compromise of the password locker can lead to compromise of the entire network. The main difference is that now the passwords are stored on or accessed from an end-user machine that probably has Internet access, email access and one or more browsers. In other words, the passwords are now stored on the most vulnerable machines in the network.
Worse, the passwords must still be manipulated by humans so there is still a temptation to use short, human-readable strings that are subject to dictionary attacks. If we are lucky enough to have a policy that dictates strong, complex and long passwords then there is no hope of getting these typed correctly into the tools where they are used. Instead, we copy and paste them. Now the passwords hang around in memory and swap files indefinitely.
Finally, consider that if all the administrators use some kind of password locker then there is a complete copy of the set of keystore passwords for each administrator. The more administrators there are, the more copies of the password database exist. As the number of copies expands, the likelihood that one of them will be compromised increases. Add enough administrators and that probability approaches certainty. You may be thinking “But T.Rob, we’ll NEVER have that many administrators!” OK, but consider the corollary: given a small group of administrators over a long enough period of time and the probability also approaches certainty.
Now I’m not saying that there is no value at all in encrypting theapplication passwords in the configuration files, or in using password lockers or in encrypted file systems. Far from it, these are all very useful tools. What I am saying though is that when we make these decisions we must be aware of our biases and attempt to neutralize them. When it comes to encrypting passwords in configuration files we have a gut instinct which tells us that this is good and we tend to overestimate the incremental security value added. We employ a mental calculus that tells us decrypting the password is hard and that accessing files is easy when in fact the opposite is true. Any encryption on those passwords must be reversible and is usually the easy part. In fact, if I can read your .kdb files I don’t even need to reverse the password since I can fire up my own queue manager of the same name and it can use the .kdb and stash file directly. Since my queue manager knows how to read the stash file it is not the actual password that is important, just the ability to use it to impersonate you. The entirety of your application security often comes down to nobody other than administrators being able to read the configuration files of the application.
We also tend to overlook the extent to which distributed management of keystores and passwords weakens security. All of the management tools for keystores and public/private key pairs are designed to provide security with the assumption that the private key is generated where it will be used and then never moved or copied. The FIPS-approved modules and algorithms even go so far as to wipe temporary storage before releasing it back to the operating system just to prevent partial artifacts of key generation from leaking out. But no matter how secure the crypto tools are, they can’t compensate for administrators leaving a trail of passwords in copy/paste buffers and scp temporary files across the network. I once had a client who actually emailed their kdb files around. Encrypting the passwords results in a very slight reduction in risk if an attacker gains read access to the configuration files, but it does then require us to manage the passwords elsewhere. Taking that step opens up a whole new avenue of attack that may be more vulnerable than the risk we intended to mitigate in the first place.
What is the alternative? Consider a system where the keystores, private keys and their passwords are generated in place and stored locally in configuration files, in plain text. This has several advantages:
- The passwords need not be human readable. I typically use 63-character pseudo-random strings that would be impossible to brute-force. I also wrap the keystore management commands in scripts that extract the password from the configuration file and insert it into the maintenance commands as needed.
- The keys and passwords are never copied or pasted on the workstations of human users, or otherwise shipped around the network, so there is no possibility of leaving a trail of artifacts.
- There is no temptation to use the same password on multiple nodes. Compromise of one does not lead to any particular advantage in compromising others.
- Keys and passwords are stored only on machines that have tight physical and OS security and do not have access to the Internet. (None of your servers have unrestricted access to the Internet, right? RIGHT?!?!)
- There need only be one copy of any private key and keystore password. The fewer the number of copies, the less chance of compromise. If there are additional copies, these are on encrypted backup or other controlled media and not on human-user workstations.
The practice of encrypting passwords in configuration files springs from implicit assumptions that someone other than a legitimate user can read those files and that the attacker will not be able to perform the same procedure that the application uses to render the plain text password. These are very bad assumptions on which to base a security model. Always assume that anyone who can read the keystore and password file can impersonate the application, even if the password is encrypted, and design the security on that basis. First and foremost, the operating system, underlying file system permissions and application access must be locked down tight enough so that only the legitimate administrators have read or better access to the configuration files.
Where application passwords must be encrypted or obfuscated, consider how to implement password management so that it does not weaken effective security. For example, WebSphere MQ expects .kdb passwords to be stored in the stash file and there is no option for the queue manager to use a plain text version. The level of security achieved then depends on how the administrative passwords are managed and this can range from “no weaker than the OS and filesystem” to “completely exposed on an administrator’s bot-infected laptop.” Any attacker with read access to the stash file could easily reverse it so there is no additional risk incurred by storing the actual plain text password in the same filesystem with the other kdb files. Any other measures which require the password or keystore to exist on any other workstation or server node reduce the effective level of security.
To the extent that encrypting passwords in configuration files leads to the implementation of distributed password management schemes, the result is often that the overall level of security is reduced. If we know this, neutralize our instinctual biases as much as possible and then make a deliberate decision that the increased risk is worth the benefit in convenience, perhaps that is good enough. But if we encrypt passwords in configuration files because our gut instinct tells us that encrypted passwords are safer, or because our security model is to protect against the threat of non-administrators having read access to configuration files, then the result is likely to be a system that is far less secure than we believe it to be. That is usually worse than no security at all.