Cluster security

Well, the residency to write the new WebSphere MQ Security book is past the halfway point and we are working diligently to finish up on time. I’m happy to say that one of my favorite new security topics is covered in the book: cluster security.

With point-to-point networks there was always had a separate inbound channel for each remote queue manager and that allowed you to authorize each one to a different subset of queues. But convert that network to use clustering and there is only one inbound channel so the granularity of point-to-point channels was lost. Sure, you could use a security exit and a channel auto-definition exit but very few people went to the trouble of writing the exits. For better or for worse, most shops fell back to a monolithic security model for their cluster.

As of WMQ V7.1 though, the new CHLAUTH rules restore that missing granularity. It is now possible to map each cluster member to a specific security role based on its QMgr name, IP address, SSL certificate name or some combination of these. This is such great news that I included it in the conference presentations, blogged about it and dedicated part of a Global WebSphere Community presentation to the topic. I am extremely happy that we were able to dedicate a chapter in the new book to cluster security and document it at a level of detail not possible in presentations and blog posts.

The other aspect of the residency that I eagerly anticipated was the opportunity to dedicate some time to hands-on activities in the lab. We are using V7.5 of WebSphere MQ for our scenarios and it was released the week before the residency began. We want to make sure that our recommendations from previous versions still work and we want to perform “negative tests” – i.e. break stuff to see what happens. Sometimes what happens is unexpected.

Case in point, towards the end of last week we were talking about the attack where you can delete your SYSTEM.DEF.RCVR, create a sender of the same name and point it at your intended victim. If the target queue manager has an unprotected SYSTEM.DEF.RCVR you can gain access as an administrator. This is quite well known now and has been discussed many times on the list server and MQSeries.net. But one of the guys suggested that it is possible to do the same thing with SYSTEM.DEF.CLUSRCVR.

I was skeptical but the other guys assured me it would work, although none had tried it. Having the perfect lab with which to test it out, I decided to give it a try…and it worked. What I did was to whip up a fresh queue manager, delete SYSTEM.DEF.CLUSRCVR and create a SDR channel of the same name. I then pointed the SDR channel at an unsuspecting queue manager which happily accepted the connection and I succeeded in sending messages to the target QMgr.

I was floored!  I had always assumed that an unprotected CLUSRCVR would be vulnerable – but to rogue CLUSSDR channels, not SDRs. I had never guessed that a SDR channel could connect to a CLUSRCVR. I always assumed they were different enough, like client channels, that a SDR would not start one so I never tried it. The funny thing was Neil had never tried it either, but in his case this was because he always assumed it would work.

The good news is that anyone who has followed the security advice of disabling the default channels and authenticating the usable channels will not be exposed to this. Everything we’ve been talking about in the community and that I’ve been recommending over the last few years still applies and will protect the CLUSRCVR channels. On the other hand, if you have left your CLUSRCVR channels unprotected believing they were less vulnerable to spoofing than RCVR channels, this may be your wake-up call.

If you want to test this yourself, it is pretty easy to do. Just spin up two new queue manager instances and start a listener on one. Do not alter the SYSTEM.DEF.CLUSRCVR channel at all. On the other QMgr, delete SYSTEM.DEF.CLUSRCVR and create a new SDR channel by that name, with the CONNAME pointing at the first queue manager’s IP and listener. Make sure to create the XMitQ for the channel as well! If you are running V7.5 or later you’ll need to add a CHLAUTH rule or disable CHLAUTH to make this work with SYSTEM.DEF.CLUSRCVR, or you can try with a user-defined CLUSRCVR.  You should now be able to start the SDR channel and send messages to the target queue manager.

Migrating to V7.1 or later solves the problem for the SYSTEM.* channels but you still need to protect any user-defined channels.  The best way to mitigate this risk for all versions and cases is through strong authentication. The attacker can’t spoof a channel if he can’t get past the SSL handshake. As I noted earlier, the exposure of an unprotected channel is nothing new, should not surprise anyone, and all the routine mitigations will resolve the exposure here as well. It was the notion of a SDR talking to a CLUSRCVR that caught me by surprise, and that a CLUSRCVR which is not enrolled in a cluster would be operable at all. In hindsight it makes perfect sense, though.

We have another week and a half of residency to go and everyone is furiously writing. I do not have a publication date but I expect that a draft would be posted within a few weeks after we wrap up on the 20th of July. Hopefully, no more big surprises between now and then! In any case, I’ll keep you posted here.

 Update 17 July – Modified to point out differences in versions and that V7.1 and later continues to be the best option where security is a requirement.