The title of my last post, Client-based runmqsc gotcha, was a bit of a misnomer. Although I stumbled onto it using runmqsc client, the root cause of the issue really lies with the Command Server. That means anything that drives commands through the Command Server can run into this issue, making it a much bigger deal than I originally thought.
IBM has replied to the PMR so I’ll discuss that along with a suggested strategy to mitigate the risk and hopefully avoid Production outages.
Most shops running more than a handful of queue managers end up writing a script or two that sends commands to runmqsc and parses the responses. Until recently the options were to run those scripts locally or to access remote queue managers using the MQ network. Neither of these options is ideal. Use of local scripts requires maintaining a code base distributed across a variety of hosts and often disparate platforms. Sending runmqsc commands across the MQ network solves those problems but requires a security model in which at least some of the adjacent queue managers can administer each other.
The new MQ Client capabilities in runmqsc give us the best of both worlds. We can maintain a single set of scripts in a central location and use them to connect directly to each of the queue managers using MQ client. Some functions that require correlation of state across multiple queue managers are actually easier to build using MQ client.
There is, however, a gotcha that anyone using client-based runmqsc should be aware of. I’ll explain that here along with some techniques to mitigate the issue and feedback from the PMR I opened.
Every now and then we hear that a cipher algorithm has fallen to a new cracking technique. This cascades into a new round of deprecating any ciphersuites that rely on the newly cracked algorithms. Over the years we’ve moved from SSL to TLS, from DES to 3DES, from MD5 to SHA, and so on. The list of items on MQ’s Deprecated Ciphers page grows slowly but relentlessly over time.
But not all of the ciphersuites that IBM has deprecated are broken. So why are they on the list and should we use them? This post will attempt to shed some light on those questions.
A growing number of my clients are deploying IBM MQ on Amazon EC2 instances and a common need that I see emerging is for instrumentation and tooling. When the MQ instance is ephemeral, deploying instances on demand and decommissioning just as suddenly, lots of things the MQ Admin used to do by hand need to be automated. This includes build-time things such as defining objects, run-time tasks like enabling or disabling queues in the cluster, and forensic capabilities such as archiving error logs.
It is this last item that concerned a recent customer. Their main requirement was to ingest MQ error logs in real time, or at least close to it, so those logs would survive death of the virtual host on which they were generated. Getting Splunk to ingest the logs was ridiculously easy. Just define the log files as a Splunk data input and immediately they become available through the Splunk search interface.
That’s all well and good if all you want to do is browse through the logs or search them for particular error codes. To get the benefit of Splunk analytics requires the error logs to be parsed into fields. Then instead of merely searching for error codes you already know about, you can ask Splunk to show you a report of all the error codes sorted by prevalence, by frequency over time, or even which ones are the rare outliers. All the analytic capabilities are usable once the fields are parsed. Better yet, parse logs from many queue managers and now you can spot trends or pick out nodes that are showing early signs of distress. That’s really useful stuff and Splunk provides it right out of the box, but only for log types it knows how to parse. So let’s teach it to parse IBM MQ error logs, shall we?
After I tweeted a link to an IBM blog post on how to start and stop IBM MQ using systemd, an IBMer responded to say “it surprised me to hear that some #IBMMQ customers have to manually restart their QMs when the box comes up.”
My reply was brutally frank: “Should be no surprise – the serviceability gap with MQ start/stop API resembles an open-pit mine. As a result most shops either don’t do it well, or don’t do it at all. Mortgage payments on the technical debt needed here desperately.”
Not wanting to leave that hanging out there with no explanation, this post describes in excruciating detail what’s wrong. Hopefully, that’s the first step to getting it fixed.
The theme of my sessions at this year’s MQTC (and hopefully also at IBM Think if they are accepted) is cloud and virtualization, if you are reading the abstracts. If you come to the session you find it’s really about designing architecture around configuration management and tools with the specific intent of driving administrative overhead burden and defects down to near zero. So it was a bit distressing yesterday when during the demo a string of errors cascaded across the screen. Unless you are into schadenfreude, in which case watching my live demo auger into the ground might have been fun for you. But in the end, the event more proves my point rather than invalidating it. Here’s why.
My two sessions from this year’s MQTC are posted:
MQ Automation: Config Management Using Baselines, Patterns and Apps
Take the grunt work out of MQ configuration management for virtualization, cloud, and large networks by applying a layered approach. This session will introduce the concept of building an MQ configuration from a baseline, then defining a class of service with a pattern layer, and finishing off with application configurations. This modular approach dramatically improves consistency, quality, and flexibility while greatly reducing cost. In compliance environments it provides a definitive as-specified configuration to which the as-running state can be reconciled at intervals or in near-real time. A basic script framework for implementing this system will be reviewed as well.
MQ Automation: Config Management using Amazon S3
The central server needed to set up an MQ configuration Management system turns out to be a consistent showstopper, but with a few pennies and a few scripts you can use Amazon Simple Storage. This session introduces scripts that automate QMgr builds with a local shell script that queries a flat-file configuration database stored in the cloud. It’s dirt cheap and super simple yet can reduce the time and cost of building MQ nodes while improving quality and consistency.
Note: I created a dedicated user for the conference and am supplying the ID and key in the session materials. Download the slides so you can cut-and-paste the commands to install the AWS metadata files.
In case you hadn’t noticed yet, IBM has quietly changed the format of the stash file so that the various unstash programs no longer work. In this post I’ll discuss some of the security implications of that change and, since I never quite grew up, also channel Sean Penn’s Spiccoli from Fast Times at Ridgemont High and make a lot of stash jokes. As Spiccoli might say, “Dude, IBM broke my stash!”
The security maturity progression in MQ starts with access control. First we isolate MQ Admin access, then add granular user and application access. This class of security control is known as intrusion prevention. After mastering that the next phase includes stronger accountability and intrusion detection. These typically include enabling event messages and archiving security-relevant events from the event queues and the error logs.
In order to detect security-relevant events and hold people strictly accountable within the access roles defined, the product itself must be further along that continuum than the customer is. In particular, the intrusion detection controls must report all events and do so accurately and these tests show that to not be true in all cases.
The issues reported in this post affect the ability of an administrator or auditor to accurately detect security-relevant events and enforce accountability. Since error logs roll over, archiving them is subject to a window within which error logs might roll off and be irretrievably lost. Many shops rely on event messages to provide a level of assurance acceptable to auditors and security-conscious users and these errors show that reliance may be misplaced.
I’ve added a “Versions” tab to the results matrix, corrected some copy/paste errors, and uploaded new copies of the PDF and Excel versions. Over time as new results are added or corrections made I’ll replace the existing documents so the links do not change. These are active documents so expect changes frequently.
I won’t post updates to the GitHub documents since these will probably be the most active artifacts of the entire project – and because GitHub shows you the complete history. Much thanks to fjbsaper and Josh McIver for updates and edits on the tools.
Stack Exchange MQ Q&A