MQ Admins are getting serious about TLS channels these days, but it isn’t always easy because there’s a fairly steep learning curve. Though there is plenty of documentation for the MQ aspects, and for X.509 and TLS itself, very little exists that translates these requirements into a procedure you could actually use to provision a certificate signed by a commercial CA. The Certificate Authorities document the provisioning process for certs to be used in the various web servers, but the thing you need a certificate for isn’t a web server the CA-provided documentation is often lacking. In particular, official documentation from the CAs about MQ certs is almost non-existent.
To address that gap I wanted something that showed the process, start to finish, of enabling TLS onto an existing pair of SDR/RCVR channels. I don’t know about you but I personally need to understand a process from a high-level in order to best understand how all the pieces fit together and their up- and downstream dependencies. This is that high-level overview.
The first delivery from this project is a 6-part video tutorial showing the provisioning of CA-signed certificates for MQ and configuration of two queue managers using TLS channels. The series takes under an hour beginning with no queue managers, walks through generation and submission of the CSR, receiving the signed cert, and finishes up with two queue managers using mutually authenticated TLS channels, successfully tested by sending messages.
I used GoDaddy because they agreed to give me some certs for free to do the project but what I found is that they have REALLY good cert management tools. For example, they let you re-key a cert – which is good because as I was doing the dry run I discovered a GSKit bug that blows away your KDB, and any certs that are in it. I generated a new CSR with the same DN, uploaded it , and got back a new cert based on the new key pair, but functionally equivalent to the one I’d lost. THAT is really cool.
Divide and conquer
In the tutorial, and for all of my consulting engagements involving setting up MQ TLS channels, I prescribe a step-by-step implementation:
- Start with working channels.
- Server-only authentication.
- Mutual authentication.
- Filter the Distinguished Name using CHLAUTH, SSLPEER or exits.
Many people go the route of getting certificates provisioned to all the queue managers and enabling mutually-authenticated TLS channels without going through the intermediate step of server-only authentication. The problem with this is that it becomes more difficult to determine which side might be causing the issue because the handshake is bidirectional.
Starting with server-only authentication (where the queue manager with the inbound channel has a personal certificate but the other one does not) the handshake goes in only one direction, making diagnostics much simpler.
The same goes for certificate filtering. Without filtering, the presence of the CA root and intermediate certificates in the KDB effectively creates an “allow all” policy. The queue manager will accept any certificate the CA has ever or will ever sign using that signer chain, so long as it has not expired and is otherwise valid. Obviously that is far too permissive and needs to be locked down. Just don’t do that until all the prerequisite parts of the connection are up and running.
Server-Only vs. Mutual Authentication
This is one of the places where people frequently get hung up so it is worth a few moments of dedicated discussion. The first thing to know is that the TLS server, which is the queue manager receiving the connection request, always authenticates itself by presenting a personal certificate to the thing requesting a connection. If it cannot open its KDB or cannot find its certificate, it refuses the connection request and closes the connection.
Though the server always authenticates itself, authentication of the client is configurable. It is the SSLCAUTH channel attribute that determines the authentication policy of the queue manager receiving the connection request. Set that to REQUIRED and your queue manager will reject any connection request that does not present a certificate. Set it to OPTIONAL and the client is not obligated to present a certificate in order to connect. However, if a certificate is presented, it must be valid.
The subtlety in this configuration is that the behavior on the client side is not impacted by the SSLCAUTH setting. If the client can find a personal certificate, it presents that certificate during the connection handshaking. Many hours have been spent trying to debug what was assumed to be server-only authentication, only to find out that the problem in the channel had to do with validation of the client’s certificate. That is why in the final video you will see that I temporarily remove the personal certificate from the client-side queue manager for testing server-only authentication, then later restore it in order to test mutual authentication.
Another trap to avoid is applying the same tests in both directions. For example, when setting up my two queue managers ASH and BIRCH (my queue managers all have tree names), I used a SDR on ASH and a RCVR on BIRCH for all testing. Many people believe that they must test the SDR channels on both sides but doing so is a treacherous path. Here’s why.
The queue manager with the inbound channel (that’s any channel of type RCVR, RQSTR, CLUSRCVR or SVRCONN) must have a personal certificate. For the initial phase of testing the queue manager or client on the opposite side must not have a personal certificate. If you were to switch the roles of the client and server, the node that didn’t need a certificate in the first phase would now need one. It would be necessary to add its personal cert and test with a different set of channels other than the ones tested with in the first test.
While it is necessary at some point to add the personal cert on the client side, it can be tested using the exact same set of channels and with minimal other changes, simply by setting SSLCAUTH(REQUIRED) on the receiver and re-driving the same SDR that was testes for server-only authentication.
All of the client-side validation that we need is performed either way. The difference is that one method uses the exact same set of channel objects for all testing, and therefore cuts the number of possible failure points by 50%. So use the same channels for all testing of the TLS configuration. Once you get mutual auth in one direction, all of the certificate and eystore validation required to get it going in the opposite direction has been completed. If there’s an error setting up the return channels, the only possibility is something mechanical such as a typo in the channel configuration.
The video tutorial is a bit under an hour and it took only about two hours to produce. Most of the time you don’t see in the video was taken up either waiting on certificate validation or diagnosing and replicating the GSKit bug I discovered. The process of setting up TLS channels is not quite as easy as it looks in the videos, but it isn’t as bad as people often think.
But there’s a plateau at which many shops arrive where they learn the mechanical process of deploying the certificates but do not necessarily understand all of the the dependencies in the certificate-based security architecture. This is the “knowing enough to be dangerous” phase. It is very easy at this stage to deploy TLS across the messaging network, have every channel either encrypted or locked down, and still leave remote administrative access wide open. The problem with this is that, after having gone through the security implementation project, there can be a great reluctance to question its efficacy or test it. That leads to a false sense of security which sometimes doesn’t go away until an adverse event demonstrates the exposure.
My strong recommendation for shops that are newly embarking on a TLS deployment is to get some outside help from a well-qualified specialist. Hopefully, that person is me, but whoever it is should work shoulder-to-shoulder with your MQ administrators to walk through the implementation process several times, making sure to throw them the occasional curve ball so they learn how to recognize and diagnose problems. If the engagement ends with a fully configured TLS network that your administrators can’t operate, the engagement has been less than successful. The MQ admins need to be able to synthesize novel ad-hoc solutions to issues that aren’t on the cheat sheet, and to do so in confidence that the result does not break the security model.
This is the purpose of the MQ TLS Quick Start engagement which is a specialty of mine. If you already have some TLS we can structure the engagement as an assessment to make sure all the bases are covered, or even do some live penetration testing. Let’s talk soon about what kind of services you might require. Feel free to email me at firstname.lastname@example.org or call +1 704-443-TROB (that’s +1 704-443-8762) to discuss your needs.
I’m a 1-man consultancy so if you call and I do not answer, please leave a voice mail. I’d like to be able to take every call, but that would ruin my credibility as a specialist in asynchronous messaging.
I posted a link to the playlist above, but for completeness, here is the entire list:
You can go to the playlist of all 6 videos or pick them off the menu:
- Process overview
- Configuring the baseline environment
- Generate the Certificate Signing Request
- Process the CSR at the CA web site
- Receive the certificate into the KDB
- Configure MQ for TLS
Be sure to let me know in the comments what you think, and leave suggestions for additional content. You can also subscribe to me on Slideshare and YouTube to get announcements of new slides or videos, or subscribe to this blog to get announcements for all new content.