A growing number of my clients are deploying IBM MQ on Amazon EC2 instances and a common need that I see emerging is for instrumentation and tooling. When the MQ instance is ephemeral, deploying instances on demand and decommissioning just as suddenly, lots of things the MQ Admin used to do by hand need to be automated. This includes build-time things such as defining objects, run-time tasks like enabling or disabling queues in the cluster, and forensic capabilities such as archiving error logs.
It is this last item that concerned a recent customer. Their main requirement was to ingest MQ error logs in real time, or at least close to it, so those logs would survive death of the virtual host on which they were generated. Getting Splunk to ingest the logs was ridiculously easy. Just define the log files as a Splunk data input and immediately they become available through the Splunk search interface.
That’s all well and good if all you want to do is browse through the logs or search them for particular error codes. To get the benefit of Splunk analytics requires the error logs to be parsed into fields. Then instead of merely searching for error codes you already know about, you can ask Splunk to show you a report of all the error codes sorted by prevalence, by frequency over time, or even which ones are the rare outliers. All the analytic capabilities are usable once the fields are parsed. Better yet, parse logs from many queue managers and now you can spot trends or pick out nodes that are showing early signs of distress. That’s really useful stuff and Splunk provides it right out of the box, but only for log types it knows how to parse. So let’s teach it to parse IBM MQ error logs, shall we?
After I tweeted a link to an IBM blog post on how to start and stop IBM MQ using
systemd, an IBMer responded to say “it surprised me to hear that some #IBMMQ customers have to manually restart their QMs when the box comes up.”
My reply was brutally frank: “Should be no surprise – the serviceability gap with MQ start/stop API resembles an open-pit mine. As a result most shops either don’t do it well, or don’t do it at all. Mortgage payments on the technical debt needed here desperately.”
Not wanting to leave that hanging out there with no explanation, this post describes in excruciating detail what’s wrong. Hopefully, that’s the first step to getting it fixed.
The theme of my sessions at this year’s MQTC (and hopefully also at IBM Think if they are accepted) is cloud and virtualization, if you are reading the abstracts. If you come to the session you find it’s really about designing architecture around configuration management and tools with the specific intent of driving administrative overhead burden and defects down to near zero. So it was a bit distressing yesterday when during the demo a string of errors cascaded across the screen. Unless you are into schadenfreude, in which case watching my live demo auger into the ground might have been fun for you. But in the end, the event more proves my point rather than invalidating it. Here’s why.
My two sessions from this year’s MQTC are posted:
MQ Automation: Config Management Using Baselines, Patterns and Apps
Take the grunt work out of MQ configuration management for virtualization, cloud, and large networks by applying a layered approach. This session will introduce the concept of building an MQ configuration from a baseline, then defining a class of service with a pattern layer, and finishing off with application configurations. This modular approach dramatically improves consistency, quality, and flexibility while greatly reducing cost. In compliance environments it provides a definitive as-specified configuration to which the as-running state can be reconciled at intervals or in near-real time. A basic script framework for implementing this system will be reviewed as well.
MQ Automation: Config Management using Amazon S3
The central server needed to set up an MQ configuration Management system turns out to be a consistent showstopper, but with a few pennies and a few scripts you can use Amazon Simple Storage. This session introduces scripts that automate QMgr builds with a local shell script that queries a flat-file configuration database stored in the cloud. It’s dirt cheap and super simple yet can reduce the time and cost of building MQ nodes while improving quality and consistency.
Note: I created a dedicated user for the conference and am supplying the ID and key in the session materials. Download the slides so you can cut-and-paste the commands to install the AWS metadata files.
In case you hadn’t noticed yet, IBM has quietly changed the format of the stash file so that the various unstash programs no longer work. In this post I’ll discuss some of the security implications of that change and, since I never quite grew up, also channel Sean Penn’s Spiccoli from Fast Times at Ridgemont High and make a lot of stash jokes. As Spiccoli might say, “Dude, IBM broke my stash!”
I’ve added a “Versions” tab to the results matrix, corrected some copy/paste errors, and uploaded new copies of the PDF and Excel versions. Over time as new results are added or corrections made I’ll replace the existing documents so the links do not change. These are active documents so expect changes frequently.
I won’t post updates to the GitHub documents since these will probably be the most active artifacts of the entire project – and because GitHub shows you the complete history. Much thanks to fjbsaper and Josh McIver for updates and edits on the tools.