Managing CA-signed certificates

MQ Admins are getting serious about TLS channels these days, but it isn’t always easy because there’s a fairly steep learning curve.  Though there is plenty of documentation for the MQ aspects, and for X.509 and TLS itself, very little exists that translates these requirements into a procedure you could actually use to provision a certificate signed by a commercial CA.  The Certificate Authorities document the provisioning process for certs to be used in the various web servers, but the thing you need a certificate for isn’t a web server the CA-provided documentation is often lacking.  In particular, official documentation from the CAs about MQ certs is almost non-existent.

To address that gap I wanted something that showed the process, start to finish, of enabling TLS onto an existing pair of SDR/RCVR channels.  I don’t know about you but I personally need to understand a process from a high-level in order to best understand how all the pieces fit together and their up- and downstream dependencies.  This is that high-level overview.

The first delivery from this project is a 6-part video tutorial showing the provisioning of CA-signed certificates for MQ and configuration of two queue managers using TLS channels. The series takes under an hour beginning with no queue managers, walks through generation and submission of the CSR, receiving the signed cert, and finishes up with two queue managers using mutually authenticated TLS channels, successfully tested by sending messages.

I used GoDaddy because they agreed to give me some certs for free to do the project but what I found is that they have REALLY good cert management tools.  For example, they let you re-key a cert – which is good because as I was doing the dry run I discovered a GSKit bug that blows away your KDB, and any certs that are in it.  I generated a new CSR with the same DN, uploaded it , and got back a new cert based on the new key pair, but functionally equivalent to the one I’d lost.  THAT is really cool.

Divide and conquer

In the tutorial, and for all of my consulting engagements involving setting up MQ TLS channels, I prescribe a step-by-step implementation:

  1. Start with working channels.
  2. Server-only authentication.
  3. Mutual authentication.
  4. Filter the Distinguished Name using CHLAUTH,  SSLPEER or exits.

Many people go the route of getting certificates provisioned to all the queue managers and enabling mutually-authenticated TLS channels without going through the intermediate step of server-only authentication.  The problem with this is that it becomes more difficult to determine which side might be causing the issue because the handshake is bidirectional.

Starting with server-only authentication (where the queue manager with the inbound channel has a personal certificate but the other one does not) the handshake goes in only one direction, making diagnostics much simpler.

The same goes for certificate filtering.  Without filtering, the presence of the CA root and intermediate certificates in the KDB effectively creates an “allow all” policy.  The queue manager will accept any certificate the CA has ever or will ever sign using that signer chain, so long as it has not expired and is otherwise valid.  Obviously that is far too permissive and needs to be locked down.  Just don’t do that until all the prerequisite parts of the connection are up and running.

Server-Only vs. Mutual Authentication

This is one of the places where people frequently get hung up so it is worth a few moments of dedicated discussion. The first thing to know is that the TLS server, which is the queue manager receiving the connection request, always authenticates itself by presenting a personal certificate to the thing requesting a connection.  If it cannot open its KDB or cannot find its certificate, it refuses the connection request and closes the connection.

Though the server always authenticates itself, authentication of the client is configurable.  It is the SSLCAUTH channel attribute that determines the authentication policy of the queue manager receiving the connection request.  Set that to REQUIRED and your queue manager will reject any connection request that does not present a certificate.  Set it to OPTIONAL and the client is not obligated to present a certificate in order to connect.  However, if a certificate is presented, it must be valid.

The subtlety in this configuration is that the behavior on the client side is not impacted by the SSLCAUTH setting.  If the client can find a personal certificate, it presents that certificate during the connection handshaking.  Many hours have been spent trying to debug what was assumed to be server-only authentication, only to find out that the problem in the channel had to do with validation of the client’s certificate.  That is why in the final video you will see that I temporarily remove the personal certificate from the client-side queue manager for testing server-only authentication, then later restore it in order to test mutual authentication.

Bi-directional testing

Another trap to avoid is applying the same tests in both directions.  For example, when setting up my two queue managers ASH and BIRCH (my queue managers all have tree names), I used a SDR on ASH and a RCVR on BIRCH for all testing.  Many people believe that they must test the SDR channels on both sides but doing so is a treacherous path.  Here’s why.

The queue manager with the inbound channel (that’s any channel of type RCVR, RQSTR, CLUSRCVR or SVRCONN) must have a personal certificate.  For the initial phase of testing the queue manager or client on the opposite side must not have a personal certificate.  If you were to switch the roles of the client and server, the node that didn’t need a certificate in the first phase would now need one.  It would be necessary to add its personal cert and test with a different set of channels other than the ones tested with in the first test.

While it is necessary at some point to add the personal cert on the client side, it can be tested using the exact same set of channels and with minimal other changes, simply by setting SSLCAUTH(REQUIRED) on the receiver and re-driving the same SDR that was testes for server-only authentication.

All of the client-side validation that we need is performed either way.  The difference is that one method uses the exact same set of channel objects for all testing, and therefore cuts the number of possible failure points by 50%.  So use the same channels for all testing of the TLS configuration.  Once you get mutual auth in one direction, all of the certificate and eystore validation required to get it going in the opposite direction has been completed.  If there’s an error setting up the return channels, the only possibility is something mechanical such as a typo in the channel configuration.

TLS Quick-Start

The video tutorial is a bit under an hour and it took only about two hours to produce.  Most of the time you don’t see in the video was taken up either waiting on certificate validation or diagnosing and replicating the GSKit bug I discovered.  The process of setting up TLS channels is not quite as easy as it looks in the videos, but it isn’t as bad as people often think.

But there’s a plateau at which many shops arrive where they learn the mechanical process of deploying the certificates but do not necessarily understand all of the the dependencies in the certificate-based security architecture.  This is the “knowing enough to be dangerous” phase.  It is very easy at this stage to deploy TLS across the messaging network, have every channel either encrypted or locked down, and still leave remote administrative access wide open.  The problem with this is that, after having gone through the security implementation project, there can be a great reluctance to question its efficacy or test it.  That leads to a false sense of security which sometimes doesn’t go away until an adverse event demonstrates the exposure.

My strong recommendation for shops that are newly embarking on a TLS deployment is to get some outside help from a well-qualified specialist.  Hopefully, that person is me, but whoever it is should work shoulder-to-shoulder with your MQ administrators to walk through the implementation process several times, making sure to throw them the occasional curve ball so they learn how to recognize and diagnose problems.  If the engagement ends with a fully configured TLS network that your administrators can’t operate, the engagement has been less than successful.  The MQ admins need to be able to synthesize novel ad-hoc solutions to issues that aren’t on the cheat sheet, and to do so in confidence that the result does not break the security model.

This is the purpose of the MQ TLS Quick Start engagement which is a specialty of mine.  If you already have some TLS we can structure the engagement as an assessment to make sure all the bases are covered, or even do some live penetration testing.  Let’s talk soon about what kind of services you might require.  Feel free to email me at t.rob@ioptconsulting.com or call +1 704-443-TROB (that’s +1 704-443-8762) to discuss your needs.

I’m a 1-man consultancy so if you call and I do not answer, please leave a voice mail.  I’d like to be able to take every call, but that would ruin my credibility as a specialist in asynchronous messaging.

The videos

I posted a link to the playlist above, but for completeness, here is the entire list:

You can go to the playlist of all 6 videos or pick them off the menu:

  1. Process overview
  2. Configuring the baseline environment
  3. Generate the Certificate Signing Request
  4. Process the CSR at the CA web site
  5. Receive the certificate into the KDB
  6. Configure MQ for TLS

Be sure to let me know in the comments what you think, and leave suggestions for additional content.  You can also subscribe to me on Slideshare and YouTube to get announcements of new slides or videos, or subscribe to this blog to get announcements for all new content.

This entry was posted in IBMMQ, Security, WMQ Security and tagged , , , , , , , . Bookmark the permalink.

10 Responses to Managing CA-signed certificates

  1. Prasanna says:

    Hi Rob, Thanks for the informative article. I am looking for a step by step prodecure to upgrade my IBM MQ non ssl channel to TLS1.2. I am struggling to get this done since few days but struck with 2393, 2381 MQ Errors. Kindly help me with any info or article.

  2. Ruud van Zundert says:

    Hi T.Rob. I finally had some time to investigate MQ V8 on my own home server and decided to incorporate ‘proper’ SSL security. FYI this is the set up: Windows 10 … running a vm via VMware Player … running Opensuse Leap 42.1 … 90 day evaluation version of MQ 8.0.0.4 (including mft).

    Some observations:
    – being used to gsk7cmd I looked for gsk8cmd … but this is no longer shipped with MQ V8..
    … but subsequently found this IBM link to explain it:
    http://www-01.ibm.com/support/knowledgecenter/SSFKSJ_8.0.0/com.ibm.mq.mig.doc/q001200_.htm

    – the runmqckm (non-fips version of runmqakm) created the key database, but instead of the usual 4 files (key.kdb, key.rdb,key.sth.key.crl) it did not create key.crl

    – thx for the -stashed parm … learnt something new there

    – something worth mentioning: gskit 8 no longer creates that ‘usual’ set of CAs in the key store. This is a good thing as we currently have a procedure to delete these.

    – As you already alluded to in your excellent videos, typos do occur, and I made one when I created the CSR and repeated the typo in the label when receiving the signed certificate. I was, however, disappointed that the receive did not spot my mistake, eg I added label ibmwebspremqxxx and it accepted it. I was not until I tested the SSL connection that I received the “AMQ9637: Channel is lacking a certificate” which in the first instance was a great surprise as the Root and Intermediate CAs were loaded plus the qmgr cert.
    Having blown off steam (counting to 10) and listing the contents of the keystore showed my error… at which point I (re) discovered the fact that you can rename the label… pfff

    https://www.ibm.com/developerworks/community/blogs/belinky/entry/did_you_know_you_can_rename_a_certificate_label?lang=en

    – In the process of setting everything up, I started with self-signed certs, and quickly discovered the additional work required for each certificate, so I decided to investigate a way of having my own ‘proper’ CAs. The following website was quite useful, and it helped me set up the Root and Intermediate CA… and how to take my own qmgr CSR and sign it using the Intermediate CA.

    https://jamielinux.com/docs/openssl-certificate-authority/index.html

    – you make extensive use of openssl … which itself also have weaknesses

    https://www.openssl.org/news/vulnerabilities.html

    it is worth checking the version people are using. In our own shop we were running old and new versions… turned out the old versions were still ‘ok’.

    – I saw you mentioned the use of san-dnsname. Is that really now compulsory, or is it the commercial providers that insist on it. In my set up i did not use it.

    – One thing I want to get straight in my mind is the number of suffices that seem to be available for the various files. I personally use file.csr for my own CSR requests… and the signed certificate has file.pem. In your example the suffix crt is used.

    – Finally, just to report that we too are in the (belated) throws of converting all our SSL V3 ciphers to TLS … v1.0 initially … moving to V1.2 where we have MQ V8 with gskit 8 (curr MQ 7.5.0.5).

    One historical issue we have is related to MQ on the IBM Mainframe. It turns out that the use of TRIPLE_DES_SHA_US caused the mainframe channel initiator (CHIN) to use up so much cpu that it cost the customer in excess of 100K euros… so they opted for authentication only … using NULL_SHA. That was ‘ok’ until we were bombarded by all the SSL V3 weakness warnings. So either the customer accepts the ‘weakness’ and lower cpu costs, or bite the bullit and go for TLS, or … we are investigating the use of hardware encryption.

    Sorry for the long reply… and mentioning some off-topic stuff, but I wanted to give you feed back and share my experiences and woes 😉

    Ruud

  3. Ashish Goel says:

    Hi Rob,

    The article is interesting and helpful when establishing SSL comm between two queue managers. I am configuring something similar but b/w MQ & WAS. I have created personal cert req, self signed it and extracted the self signed certificate which is copied to the WAS truststore and similarly I have created a personal cert on WAS and added it to Qmgr keystore file. Now this works as long as the SSLCAUTH is set to OPTIONAL but when changed to REQUIRED, the connection fails.

    Qmgr logs says:
    AMQ9637: Channel is lacking a certificate.

    App log says:
    CC=2;RC=2059;AMQ9503: Channel negotiation failed.

    Can you suggest the possible cause & fix. Thanks.

    • T.Rob says:

      I have created personal cert req, self signed it and extracted the self signed certificate…

      The term “self-signed” means that the certificate signs itself, not that the certificate is signed by you. The term has a very specific technical meaning. For whatever reason, the WAS instance isn’t finding its personal cert. You will need to make sure the cert is in your Key Store and not the Trust Store and check all the other configurations that tell WAS which cert to use and where it resides. You might also try using one of the MQ sample programs like amqsget or amqsput with the same cert.

  4. Pingback: MQGem Monthly (March 2015) | MQGem Software

  5. Garrett says:

    Thank you Rob, this article has been a big help to our MQ group since we’re getting ready to take the TLS plunge.

    I don’t know if this is the right forum, but I do have a question. I created a csr for my V8 qmgr and sent it to our CA, who sent me back a signed cert. When I attempt to run the runmqakm -cert -receive command, it throws an error: CTGSK3009W One or more certificates in the keystore could not be loaded.

    What would be some general causes for gskit giving this error? I’m searching Google and haven’t yet come across a sufficient answer. If you need some more specific info, let me know.

    • Buriz says:

      Had the same thing. Issue was with naming the personal certificate before executing the runmqakm -cert -receive command. Cert that you are receiving should be called something.crt – the .crt seems to be important for MQ.

    • T.Rob says:

      Wish I’d spotted this earlier. When receiving a CA signed cert, GSKit will try to validate it. That requires the signer chain to be loaded. Typically this error happens because the wrong signer chain has been loaded, or no signer chain is present for that CA.

  6. Mikhail says:

    Hi Rob!
    It’s very interesting post, but may be you highlight in next part a migration path to new certificates (pesronal and CA) frome expired ones.
    As for me, it’s not very easy and need to interrupt of MQ working process.

    • T.Rob says:

      Thanks, Mikhail! Have you found my previous article on renewing certificates? At least for the CA-signed ones it is fairly easy to issue a new cert signing request and submit it for renewal. Check it out here and let me know if it helps.

Leave a Reply to Ruud van ZundertCancel reply