Time for MQ to get serious about instrumentation and admin. Again.

Outstanding RFEs and feature requests have been a hot topic on the MQ list server of late.  Looking at the RFEs that have been posted and discussed, there’s a general architectural requirement many of them seem to have in common: Better support for administration and auditing.

It’s tough to ding IBM for lack of instrumentation in the product and I remember well a concerted and very public campaign to gather user experience feedback a few years back. There is considerable instrumentation in the product and that’s a Good Thing. Thanks, Hursley team and MQ management!

However, it is only recently that MQ users have been enabling security at scale, and many of the new security features are driving usage pattern changes. Much of the demand for instrumentation stands apart from security, but much of it is directly related and as the security implementations ramp up, previously latent requirements for instrumentation and administration become newly visible and in that light many gaps have emerged.  The need is urgent based on rapidly evolving market requirements and both customers and IBM will need to reevaluate their enhancement priorities.  We can’t assume priorities carry over from the last release.

Just as MQ approached what might be considered a well-developed set of administrative and instrumentation function, the market requirements evolved to make those look anemic. In light of ubiquitous breaches and more stringent security requirements, MQ needs a lot more admin and instrumentation functionality if we want to do things like prove to an auditor that the system wasn’t penetrated and have any confidence whatsoever when we say that.

Given recent developments with breaches, evolving attacks, and vulnerabilities now being discovered in deep infrastructure code like OpenSSL, that is to be expected. Furthermore, I know the MQ management team are aware that these requirements are emerging, and the reasons why, because I campaigned for them during my time as MQ Product Manager.

Simply put, until we all started using certain product capabilities in earnest, and auditors learned a little about MQ, and we all got really interested in securing the network, we didn’t know what worked and what didn’t and it wasn’t possible for IBM to know the nature of the unmet requirements. IBM got us close enough to expose those requirements but until we looked over that horizon by way of implementation, they were invisible.

One of my MQTC sessions is “Beyond Intrusion Prevention” which talks among other things about post-incident forensic analysis capability. That’s what the RFE and comment on it in the recent list server emails address. The reason for the session dovetails into this discussion – the majority of security implemented in MQ today is intrusion prevention. As a community we haven’t comprehensively looked at intrusion detection, included security breaches in our DR plan scenarios, designed architectures around post-breach business continuity, or pondered how we might do deep forensic analysis on MQ, hence the product features to support these have remained mostly latent requirements.

Among the toughest challenges with MQ and perhaps the most historically visible has been accountability. No surprise then that it is the theme of many of the outstanding RFEs. When the auditor shows up we can demonstrate all the security we’ve configured but proving it wasn’t temporarily disabled is difficult at best, too often impossible.

The kinds of enhancements to support accountability include things like:

  • The ability for an application using MQ AMS to make assertions as to the level of protection policy and throw a fatal error if those are not present.
  • The ability to inquire on staleness of the KDB.
  • The ability to set the KDB permissions to read-only at run time.
  • Default policy that disallows self-signed certs with the IsCA=Yes flag set presented as personal certs.
  • Publication of events to topics instead of queues (harder to prevent their being received by the legitimate monitoring agents).
  • A “secure mode” for the QMgr in which certain configurations cannot be changed without shutting down all channels or possibly a total bounce.
  • Resolve the conflict in that making config changes under an ID other than mqm risks authorizing “staff” or other random groups, but reducing that risk by making all changes as mqm reduces individual accountability (the RFE and comments mentioned earlier).
  • Notifications of new error log extents. We can’t all write instrumentation to watch the file system to catch error logs as they roll over and all be relatively successful at it. File system monitoring is platform-specific and in some cases very unreliable. Event messages are platform agnostic and extremely reliable.
  • Make Advanced Message Security work for the administrative interfaces of all the layered products, beginning with Broker and MFT.  It is crazy to add AMS protection to the file transfer or broker data flows but not the administrative interface.

My pet peeve for accountability is so bad I can’t lump it in as a bullet point:

It’s not possible using IBM’s documented and supported configuration for Broker to *not* be an MQ admin. Since there is no enforceable application isolation in Broker, that means EVERY FLOW is potentially an MQ admin through privilege escalation.

So when the auditor shows up and asks who has MQ admin and you run Broker, how do you answer? Is your code review so thorough that know for certain that nothing is escalating to admin privileges? Do you really want to be in a position that proving accountability means defending your code review procedures? If someone did use the broker’s privilege escalation to gain admin of the QMgr, would it be detected during the event? Could it be detected after the event through forensic analysis?  Or are you just hoping the auditor doesn’t know this about Broker?

Even though I’ve been around MQ for 20 years and believe IBM’s done a great job at adding instrumentation and admin features over the years, and despite the massive implementation of security in recent versions, I’m going to risk sounding a but unappreciative or ungrateful here.

Simply put, there’s a ton of unmet requirements in MQ that remained latent until data breach events became ubiquitous and the priority of securing the internal network, proactively detecting breaches, recovering from incidents, and tracing the sources of any incidents was elevated. People used to worry about “organized crime” but cyber-crime is way beyond organized. It’s automated, artificially intelligent, and thanks to malvertising, it is ridiculously well funded.  Meanwhile, MQ doesn’t even have an option to prevent it from overwriting its own error logs.

We should be designing our security around the principle that there are those who have been breached and those who will be breached. When attacks are automated, all targets are attractive.  No company is too insignificant to escape attack. From that perspective, it’s a huge problem when the only means to demonstrate accountability in an ESB is deep inspection and defense of your code review process or, worse, having to scan all the code as part of the audit.  Every time you are audited.

The reality of MQ in a world of ubiquitous data breaches is that it is sorely deficient in instrumentation and admin capabilities that support accountability, forensic analysis, and provide the reduced TCO that would free up MQ admin time from firefighting and triage so we can all worry about these strategic issues.

Don’t wait for IBM to get out in front of these requirements.  They might, but your best bet as an MQ user is to assume they will be reactive and provide them with something to react to.  Have a look at the RFEs.  Vote on the ones you want and, most important, add your use cases in the comments.  Don’t make a new RFE if something close exists because you will split the demand and the votes, making it less likely either will be considered.  Instead add your use case to the existing RFE as a comment.

Here are all the ones I could find that were security-related.  I did this in July and updated today with all the ones that I found on the list server recently.  Let me know if I’ve left any important ones off.  Also, I’ll be submitting some that I mentioned in my list above that aren’t yet RFEs, as well as a few others, so there will be updates to the list.

Want this as a wiki or separate document?  Let me know.

77154 Enhance user id auditing for configuration events on Unix systems that are generated by the mqm ID
76014 Provide REST API for MQ administration
76798 Enable a default local address to be specified at queue manager level
74622 Individual error logs files for Security related errors in MQ QMGRS
73848 QMgr event for stale messages on xmitq
72918 WMQ Explorer to resolve MQ clusters defined with the same name on different env.
72900 Encryption of data and log files on MQ appliance
72254 The WMQ admin group on the Windows should be in the Active Directory
71978 Include remote port for CONNAME identification of clients
70938 Defer client connection refresh when updating SSL keystore on QM
70934 Multiple QMs in connection name for MQ IPT
69927 “MQ v8 security, support for DNS in channel auth when reverse lookup fails (DNS lookup based on IP)”
69091 Add native IBM MQ ConnectionFactory for queue managers to be used as JNDI repositories (69091)
68294 Secure view of Channel Auths
67954 Refresh agent after changes to agent.properties. Shouldn’t have to take an agent outage to change sandboxRoot
66794 Need MQCSP password protection on RHEL 5
65496 Usefull command line tool to check on certificate expiry
65304 Allow control of put-to-waiting-getter at the queue level
64926 MQIPT
61701 Option to force MQ authority checking against the queue name before its placed in the transmission queue
60870 JMS Subscriber needs its own backout queue
60860 Active Directory oriented user authentication
60064 Delivery ordering with the WebSphere MQ mesaging
60035 Stop/Start Agents from MQ Explorer or Coordinator
59540 Eliminate the need for Admin Rights to install MQ Explorer
58147 Application Activity Trace – Show start and end time of all MQ API calls to the millisecond
56440 Need advanced message security support for MQ pub/sub
56438 Urgently need support for .net managed mode advanced message security
55977 Disable Cipher Suites
55415 ms81 – MQIPT – use a ListenerAddress for command listener
54651 Websphere MQ – enhance CHLAUTH User mapping with wildcards
53970 Detect unused wMQ definitions
53559 Queue Manager Configuration Events should include OAM changes
53296 Show the correct userid in the error logs when a authentication error occurs
53133 WMQ should manage encrypted passwords within qm.ini
48773 PCI-DSS compliance concerning cipher spec configuration for MQ Server
48767 MQ Packaging on Linux (RHEL)
46745 Capture CHLAUTH TYPE(BLOCKUSER) in dmpmqcfg output
44622 Allow dedicated separate storage on a queue level basis for MQ on Windows and Unix
44262 TLS ciphersuites are not supported on the websphere MQ client
44235 SSL negotiation information
43935 Use of mq kdb keystore to establish secured SSL connections
42567 Run MQIPT as Windows Service – non Admin
42528 Have the Queue Manager produce a read only message showing all the parameters its running with
41650 Command enhancement for SSL state investigation
40878 MQINQ against clustered Queue Managers – allow proper functionality with only +inq authority
39722 Increased granularity for MQ Event Message Delivery
39587 Create AMQERR01.LOG message or generate event at circular error log switch on WMQ distributed
39586 Allow a file to be read at strmqm which could contain MQSC statements – similar to z/OS CSQINP1/CSQINP2 ddname d/s
37989 Modify crtmqm to select either queue or topic based event message handling
37081 MQIPT single route for Multi-Instance qmgrs
35178 Please add NULL_NULL support to WMQ Cipherspec selections
35062 Provide topic or subscribtion option to duplicate msg id
35059 Provide information to applications to allow them to validate that AMS was active.
35058 Expose signer and signature information from MQ AMS to applications
31177 Enhance dmpmqcfg command to provide same functionality as saveqmgr
29919 Encrypt messages using AMS with sender/receiver MCA’s
29911 Enable ETC to support both XA and CCDT with groups of qmgrs
28671 WMQ Security Enhancement – Multiple Cyphers per Channel
28410 Add security exit support to WMQ FTE
28272 qm.ini changes on parm like maxchannels should work dynamically
26625 Make MUSR_MQADMIN non-interactive on Windows
24099 Non disruptive QM SSL renewal
22337 Audit MQ Admins when putting or getting from a queue
21984 Default Expiry as a queue attribute
21308 Report Creation and last modification date/time for Authorization entry
This entry was posted in Events, IIB, News, Security, WMQ, WMQ Security and tagged , , , , , , , , , , . Bookmark the permalink.

3 Responses to Time for MQ to get serious about instrumentation and admin. Again.

  1. Hi T.Rob. I don’t know if this is important, but it’s security related: Automatic standard naming & security controls @ define time , RFE ID 77988.

  2. Neil Casey says:

    Hi T.Rob. RFE 70434 is related to AIX. I believe the correct number is 70934. (Multiple QMs in connection name for MQ IPT).

    • T.Rob says:

      Good catch Neil, thanks! I have updated it and noticed several items that were new and/or that I missed last time and added them as well.

Leave a Reply