In case you hadn’t noticed yet, IBM has quietly changed the format of the stash file so that the various unstash programs no longer work. In this post I’ll discuss some of the security implications of that change and, since I never quite grew up, also channel Sean Penn’s Spiccoli from Fast Times at Ridgemont High and make a lot of stash jokes. As Spiccoli might say, “Dude, IBM broke my stash!”
Dude, what’s a stash file anyway?
First, let’s have a look at the security model of the MQ keystore. The KDB file contains all of the certificates that the queue manager uses. Some of these will be the certificates that it trusts such as CA signer certs and self-signed certs. Some of these will be certificates the queue manager “owns” and presents to the outside world to authenticate it’s identity. These last are called “personal certificates” and are the only certificates in the keystore that contain a private key.
Among the files that constitute the keystore, the file with a .kdb extension is the container for all of the certificates. Like the certificates themselves, the keystore is wrapped in a layer of cryptography. The stash file holds the password that unlocks the KDB so that the queue manager or the admin commands can use it. If there’s no stash file you are going to have a really bogus day.
Your stash is showing
The security model for the kdb and stash files is based on the fact that for either file to be useful it must be possible to reverse the encryption and render the contents of the file in usable form. If it is possible for MQ to do so then it is possible for anyone to reverse engineer the encryption. In fact, the unstash.pl script and its variants that have been floating around the Internet for many years serve exactly that purpose for files using the old stash file format. Someone with the unstash.pl program can find out what the password is on the old stash files. Woah, your stash is showing! Serious bummer, dude.
Or is it?
The first question to ask is what’s the threat of a working unstash.pl program? In order to use it, an attacker has to first be able to read the stash file and its parent directory. That means the attacker can obtain the stash and entire keystore bundle. Even if they don’t know the password, they don’t need to. They can drop the files into their own queue manager and since it knows how to parse stash files and keystores the attacker can spoof the real queue manager’s identity. Or they can run GSKit commands against it to dump the certificates, including the personal cert with its private key/
In the digital world to see your stash is to be able to steal your stash, dude. That’s like snapping your board in two on a gnarly wave and that could ruin your whole day.
So, like, why bother with a stash file, anyway?
For operational purposes the string of binary characters that makes up the stash file is the password. An attacker doesn’t need the plaintext password to open the keystore or impersonate the queue manager, they just need the file itself.
Encrypting the stash file is not about secrecy. It’s about operational readiness. Unless you want the queue manager to hang and wait for a human to type the password every time it starts,it needs a way to get at the unencrypted password and for most of us that’s the stash file. The purpose of encrypting the stash file is to allow MQ to verify its integrity each time it is opened. That’s it.
So what, exactly is the risk if the plaintext password can be known by an attacker? Well, it’s that they know the password. Is that in itself dangerous knowledge? Only if the password is reused in a context where humans have to know it and enter it interactively. You don’t want the keystore password to be the same as the mqm account password, for example. If that’s a real security threat in your shop I have a better way to address it: don’t reuse passwords in general and in particular don’t mix interactive and operational passwords.
Any password that must be decryptable can be known. That threat can never be adequately addressed simply by increasing the difficulty of decrypting the password. That threat can be addressed just by making sure that password is not valuable in any other context. Don’t use MQ keystore passwords for login accounts, the app server, the database, etc. Whoa dude, that’s heavy.
You saying I’ve been risking my stash all this time, dude?
Back in the day, the operator was required to use the -pw option on all of the GSKit commands and type the password into the command line or the program prompt. From a security perspective this was horrible. If you wanted to script the operations then the password had to be on the command line, which made it show up in plaintext in the process table when the command ran.
This approach also forced people to observe some sort of password management discipline in order to get the password when they needed to perform maintenance on a keystore. As is entirely predictable in this situation, in most cases this entailed wide reuse of passwords, improper sharing of passwords in emails and sticky notes on monitors, and lots of other practices which were terribly unsecure but which reduced the cost of administrative overhead to something the MQ admin could live with.
Even in the best case scenario in which keystore passwords were never reused in other contexts, the need for any password management manifests as bad password management in many cases, and that erodes the overall security discipline of the shop.
Woah, no way dude! Yes, way.
What am I supposed to do with my stash?
In the new world of GSKit and MQ all of the keystore commands offer the -stashed option so the administrator never needs to know the plaintext password. The commands unwind the stash file using the same method that the queue manager does. Since, as we noted earlier, to read the stash file is to be able to exploit it, there is zero additional risk imposed on the operations side of the security model. But by eliminating the need for really bad password management discipline, the security on the human process side is greatly improved.
Here’s what I do:
- Remove group and world permissions to the keystore directory and files.
- Change all my scripts and human processes to use the -stashed option for all commands other than creating the keystore.
- Generate random passwords for all existing keystores and any new ones.
- Remove all MQ password content from enterprise password vaults, sticky notes, cheat sheets, and so on.
Going down the list, the first thing is to make sure that mqm (or the platform equivalent) is the only account with access to the keystore directory and files. This is one case where the files need to be maintained by the MQ service account so I don’t even allow group access to the files if shop policies allow me to restrict them that far. Unfortunately, the MQ account needs write access to the files at run time so you can’t take the approach of write-locking them at run time and granting write permission only during maintenance.
The second bullet is really about eliminating the need for humans to know or care what the keystore password is. We can change the human processes easily enough but sometimes it’s tough to track down all the scripts that depend on using the -pw option or on operator entry of passwords.
The 3rd task is about ensuring that no queue keystores are accessible using passwords from the password vault or any of the really bad password management that might have been employed. In the case of bad password management it also means an attacker who does obtain the keystore can’t look at the password and gain any insight. For example, if the password is [companyname][current year] there’s a pretty good chance MQ wasn’t the only platform using this derivation. If the password is 64 random characters, the attacker might assume the shop uses really good password discipline and leave befor discovering that all the databases and app servers use [companyname][current year] as their passwords.
After all that’s done, remove all evidence of MQ passwords from cheat sheets, sticky notes, password vaults and anywhere else they were stored. Even old, stale information about passwords is useful to an attacker and cleaning that up is just good security hygiene.
Gnarly! IBM is cool after all
Right on, dude! If the new functionality causes a problem it isn’t that IBM broke your stash. Depending on knowing the stash file password and building a security model around that was the problem all along. When unstash.pl stopped working all that did was alert you to the problem and in the end that’s a Good Thing. Thanks, IBM!
Keystore maintenance typically doesn’t have a lot of scripts and processes so it should be easy enough to go through the instrumentation and human processes to eliminate dependency on knowing the passwords. But if you want some help with that, it’s the kind of thing I love to work on and I have some availability coming up. Surf’s up!
“… If the new functionality causes a problem it isn’t that IBM broke your stash…” I would beg to differ. I have generated around 200 certificates recently using the v9 ikeyman…. which is now causing an error on V8 systems 🙁 . It was only after I saw this article, and then Googled some more, that I found that IBM has changed the format… and if I wanted to keep the old format I could use an override ‘-v1stash’ ( http://www-01.ibm.com/support/docview.wss?uid=swg27045004 ).
What would have been much much better is if IBM had updated its code when it updated the stash file layout to automatically recognise th format of the stash file (old or new)…
Cheers … Ruud
Pingback: MQGem Monthly (July 2017) | MQGem Software