Time for MQ to get serious about instrumentation and admin. Again.

Outstanding RFEs and feature requests have been a hot topic on the MQ list server of late.  Looking at the RFEs that have been posted and discussed, there’s a general architectural requirement many of them seem to have in common: Better support for administration and auditing.

It’s tough to ding IBM for lack of instrumentation in the product and I remember well a concerted and very public campaign to gather user experience feedback a few years back. There is considerable instrumentation in the product and that’s a Good Thing. Thanks, Hursley team and MQ management!

However, it is only recently that MQ users have been enabling security at scale, and many of the new security features are driving usage pattern changes. Much of the demand for instrumentation stands apart from security, but much of it is directly related and as the security implementations ramp up, previously latent requirements for instrumentation and administration become newly visible and in that light many gaps have emerged.  The need is urgent based on rapidly evolving market requirements and both customers and IBM will need to reevaluate their enhancement priorities.  We can’t assume priorities carry over from the last release.

Just as MQ approached what might be considered a well-developed set of administrative and instrumentation function, the market requirements evolved to make those look anemic. In light of ubiquitous breaches and more stringent security requirements, MQ needs a lot more admin and instrumentation functionality if we want to do things like prove to an auditor that the system wasn’t penetrated and have any confidence whatsoever when we say that.

Given recent developments with breaches, evolving attacks, and vulnerabilities now being discovered in deep infrastructure code like OpenSSL, that is to be expected. Furthermore, I know the MQ management team are aware that these requirements are emerging, and the reasons why, because I campaigned for them during my time as MQ Product Manager.

Simply put, until we all started using certain product capabilities in earnest, and auditors learned a little about MQ, and we all got really interested in securing the network, we didn’t know what worked and what didn’t and it wasn’t possible for IBM to know the nature of the unmet requirements. IBM got us close enough to expose those requirements but until we looked over that horizon by way of implementation, they were invisible.

One of my MQTC sessions is “Beyond Intrusion Prevention” which talks among other things about post-incident forensic analysis capability. That’s what the RFE and comment on it in the recent list server emails address. The reason for the session dovetails into this discussion – the majority of security implemented in MQ today is intrusion prevention. As a community we haven’t comprehensively looked at intrusion detection, included security breaches in our DR plan scenarios, designed architectures around post-breach business continuity, or pondered how we might do deep forensic analysis on MQ, hence the product features to support these have remained mostly latent requirements.

Among the toughest challenges with MQ and perhaps the most historically visible has been accountability. No surprise then that it is the theme of many of the outstanding RFEs. When the auditor shows up we can demonstrate all the security we’ve configured but proving it wasn’t temporarily disabled is difficult at best, too often impossible.

The kinds of enhancements to support accountability include things like:

• The ability for an application using MQ AMS to make assertions as to the level of protection policy and throw a fatal error if those are not present.
• The ability to inquire on staleness of the KDB.
• The ability to set the KDB permissions to read-only at run time.
• Default policy that disallows self-signed certs with the IsCA=Yes flag set presented as personal certs.
• Publication of events to topics instead of queues (harder to prevent their being received by the legitimate monitoring agents).
• A “secure mode” for the QMgr in which certain configurations cannot be changed without shutting down all channels or possibly a total bounce.
• Resolve the conflict in that making config changes under an ID other than mqm risks authorizing “staff” or other random groups, but reducing that risk by making all changes as mqm reduces individual accountability (the RFE and comments mentioned earlier).
• Notifications of new error log extents. We can’t all write instrumentation to watch the file system to catch error logs as they roll over and all be relatively successful at it. File system monitoring is platform-specific and in some cases very unreliable. Event messages are platform agnostic and extremely reliable.
• Make Advanced Message Security work for the administrative interfaces of all the layered products, beginning with Broker and MFT.  It is crazy to add AMS protection to the file transfer or broker data flows but not the administrative interface.

My pet peeve for accountability is so bad I can’t lump it in as a bullet point:

It’s not possible using IBM’s documented and supported configuration for Broker to *not* be an MQ admin. Since there is no enforceable application isolation in Broker, that means EVERY FLOW is potentially an MQ admin through privilege escalation.

So when the auditor shows up and asks who has MQ admin and you run Broker, how do you answer? Is your code review so thorough that know for certain that nothing is escalating to admin privileges? Do you really want to be in a position that proving accountability means defending your code review procedures? If someone did use the broker’s privilege escalation to gain admin of the QMgr, would it be detected during the event? Could it be detected after the event through forensic analysis?  Or are you just hoping the auditor doesn’t know this about Broker?

Even though I’ve been around MQ for 20 years and believe IBM’s done a great job at adding instrumentation and admin features over the years, and despite the massive implementation of security in recent versions, I’m going to risk sounding a but unappreciative or ungrateful here.

Simply put, there’s a ton of unmet requirements in MQ that remained latent until data breach events became ubiquitous and the priority of securing the internal network, proactively detecting breaches, recovering from incidents, and tracing the sources of any incidents was elevated. People used to worry about “organized crime” but cyber-crime is way beyond organized. It’s automated, artificially intelligent, and thanks to malvertising, it is ridiculously well funded.  Meanwhile, MQ doesn’t even have an option to prevent it from overwriting its own error logs.

We should be designing our security around the principle that there are those who have been breached and those who will be breached. When attacks are automated, all targets are attractive.  No company is too insignificant to escape attack. From that perspective, it’s a huge problem when the only means to demonstrate accountability in an ESB is deep inspection and defense of your code review process or, worse, having to scan all the code as part of the audit.  Every time you are audited.

The reality of MQ in a world of ubiquitous data breaches is that it is sorely deficient in instrumentation and admin capabilities that support accountability, forensic analysis, and provide the reduced TCO that would free up MQ admin time from firefighting and triage so we can all worry about these strategic issues.

Don’t wait for IBM to get out in front of these requirements.  They might, but your best bet as an MQ user is to assume they will be reactive and provide them with something to react to.  Have a look at the RFEs.  Vote on the ones you want and, most important, add your use cases in the comments.  Don’t make a new RFE if something close exists because you will split the demand and the votes, making it less likely either will be considered.  Instead add your use case to the existing RFE as a comment.

Here are all the ones I could find that were security-related.  I did this in July and updated today with all the ones that I found on the list server recently.  Let me know if I’ve left any important ones off.  Also, I’ll be submitting some that I mentioned in my list above that aren’t yet RFEs, as well as a few others, so there will be updates to the list.

Want this as a wiki or separate document?  Let me know.