A comprehensive guide to surving the pandemic from an #IBMMQ perspective. Continue reading →
Every day I receive one or more new email memes in my inbox purporting to have the most up to date information available to keep me safe from the virus. Of course, none of these point to authoritative sources and most have at least one egregious error in them, begging the question of why anyone believes them in the first place. Worse, none of these informal memes, at least among those I’ve personally seen, deal with #IBMMQ specifically, which I think is a damned shame and one I hope to correct here and now. I have rounded up the most relevant health and safety tips for MQ available from the CDC (Centers for Data Control), Irene Ken whose daughter is an assistant Prof at Johns Hopkins, a friend who is a distant relative of Dr. Anthony Fauci, and IBM’s Viral Marketing Department. Here then I present to you my comprehensive list of IBM MQ Tips for the Pandemic. Detection Virus scan your messages! Unlike the real world in which testing requires atoms, our scanners are made out of software and indefinitely replicable. So when I say “we have enough scanners for everyone” it is provably true. Border patrol: If your internal QMgr attempts to break quarantine and contact another country, it probably has a virus. Use Advanced Message Security to hash message data. Altered messages will automatically be quarantined in the Protection Queue for their own protection. And yours. Check the temperature of your servers regularly. Viruses often cause servers to run hot. Your QMgr may have a virus if the Dead Queue fills rapidly or overflows. Prevention Use SOAP. Except for PCF, of course. But for everything else, use SOAP. When using SOAP, process messages for at least 20 ms. REST often to boost the immune system. In fact, if you can use SOAP over REST that’s even better. Mask sensitive data! And not just in the public domain. The CDC (Centers for Data Control) which has previously been ambiguous on the matter now recommends masking sensitive data at all times. (Or was that the PCI Council? I forget. ) Use short messages to help #FlattenTheQueue. Switch your listeners from TCP to UDP to avoid the handshake. Prefer stand-alone QMgrs to clustering. If you must cluster, try to maintain 6 network hops between adjacent nodes and adhere to any emergency policies mandated by the Full Repository. Space out your messages. Try to keep at least 6 ms between each one. Out of an abundance of caution 10 ms or more is even better. STAY IN YOUR QUEUE! Seriously, only essential messages should leave their queue during the pandemic. Scrub your input data for at least 20ms. Invalid user input is a common source of infection. To the extent possible, stop processing transactions on-prem and allow your QMgrs to work remotely in the cloud. The virus is sensitive to infrared wavelengths so purchase and install Infrared-360 because it has infrared right there in the name. Disable Message Expiry. It should go without saying but here it is: Message Expiry is the number one cause of messages expiring. Set persistence. Again, it should go without saying but non-persistence is the number one cause of messages lost in transit. Avoid Pub/Sub queues. With their one-to many and many-to-one relationships, Pub/Sub queues are the IBM MQ equivalent of Spring Breakers packed onto the beach in the middle of a pandemic. Avoid hot spots. Some queues seem to have custom properties that result in most or all messages in those queues expiring. Related messages that visit these queues succumb to expiry as well, so be sure to identify and quarantine such queues. General Avoid hoarding. A queue is not a database! Also, everyone else still needs to perform memory wipes, disk wipes, and flush their buffers too. Please be considerate If you print statements or other documents for mailing, consider switching to flushable paper. (See previous bullet point.) Beware of Delayed Delivery. Everything can appear to be just fine for minutes, hours or days, then a virus-laden message just pops up out of nowhere. Mythbusting Converting a message to a file does not defeat the virus. The virus CAN be transmitted in the RFH header or message properties. The so-called “Schrödinger Test” might detect the virus. Or it might not. Using Advanced Message Security to specify an enumerated list of recipients helps to limit the spread but does NOT totally prevent the virus which may have been in the payload prior to encryption. No, this is not the fault of the prior MQ Administration team. Mutual authentication does NOT prevent the spread of the virus. (Although it does greatly simplify contact tracing.) Same with CONNAUTH. Enabling Accounting and Statistics does NOT increase the spread of the virus. It merely provides accurate reporting which can appear to inflate numbers when prior reporting was understated. Seriously, when has your app team ever given you volume estimates that turned out to be correct? There is no advantage to FIFO vs LIFO. When it comes to the virus any-O is about as good as any other. Coping with the inevitable Despite our best efforts, some messages are non-persistent or set to expire. Here are some ways to deal with that when it happens. Turn on exception reporting. We cannot save all of our precious messages but we can arrange to be notified when they go. Keep the Dead Queue cleared. If it is allowed to fill it could overflow the disk partition, causing collapse of the entire system. #FlattenTheQueue Turn on linear logging and use transactions. Then you will at least have the memory of messages that are no longer queued. Use Advanced Message Security to establish non-repudiation so that nobody can credibly deny that your precious messages ever existed. Try to think of Exception Reports not as message obituaries, but rather as Confirmation Of Arrival to the messaging afterlife. Summing up Don’t believe everything you read on the Internet. Especially this blog post and random emails that land in your in-box purporting to have information that will save your life but can’t be found on any authoritative web site. For example, if you get the email claiming to be passed on by Irene Ken, whose daughter is an Asst. Prof at Johns Hopkins, maybe go to the actual Johns Hopkins web site and see what they say rather than believing the email at face value. Most relatives of actual scientists know better than to act as unofficial spokespersons so if this email were real Irene Ken’s daughter would be livid since posting a link to Johns Hopkins would have been MUCH better. Also, no actual epidemiologist would conflate antibiotics with antibodies so whoever sent this email to you believing it was true, or at least useful, needs an intervention. Feel free to be the one to intervene, we’d all be better for it. When it comes to pandemic advice, please ignore the viral memes and look for authoritative sources. Add “site:.gov” to your Google searches, for example. This will limit results to things posted on government web sites. Has the CDC waffled on important stuff? Sure. Is that a problem? Yes. But does that make them less authoritative than a viral email from some mystery person who claims to have a daughter working at an authoritative source institution and who is posting vital info that the source in question supposedly won’t? (Hint: If you answered ‘yes’ please unfollow me, remove me from your contact lists, and pray you make it through this alive.) <img fetchpriority="high" decoding="async" width="1200" height="1228" src="https://t-rob.net/wp-content/uploads/2020/04/IreneKen.png" alt="IreneKen" class="wp-image-1128" srcset="https://t-rob.net/wp-content/uploads/2020/04/IreneKen.png 1200w, https://t-rob.net/wp-content/uploads/2020/04/IreneKen-293x300.png 293w, https://t-rob.net/wp-content/uploads/2020/04/IreneKen-1001x1024.png 1001w, https://t-rob.net/wp-content/uploads/2020/04/IreneKen-147x150.png 147w, https://t-rob.net/wp-content/uploads/2020/04/IreneKen-768x786.png 768w" sizes="(max-width: 1200px) 100vw, 1200px" />
The title of my last post, Client-based runmqsc gotcha, was a bit of a misnomer. Although I stumbled onto it using runmqsc client, the root cause of the issue really lies with the Command Server. That means anything that drives … Continue reading →
The title of my last post, Client-based runmqsc gotcha, was a bit of a misnomer. Although I stumbled onto it using runmqsc client, the root cause of the issue really lies with the Command Server. That means anything that drives commands through the Command Server can run into this issue, making it a much bigger deal than I originally thought. IBM has replied to the PMR so I’ll discuss that along with a suggested strategy to mitigate the risk and hopefully avoid Production outages. How it breaks Root cause If a PCF command causes the Command Server to return more result messages than the MAXDEPTH of the reply-to queue, the Command Server may return partial result sets and silently abort further processing. Usually the client consuming the responses can get them fast enough that a result set only slightly larger than MAXDEPTH will succeed. However, as the result set size increases beyond that the failure rate increases. In my case when the Command Server was asked to return a result set of 4005 objects into a reply-to queue with MAXDEPTH(3000) the failure rate was about 60%. Differing behaviors I saw three different failure behaviors in runmqsc. In the best case runmqsc printed an “AMQ8101: WebSphere MQ error (805) has occurred” to indicate a problem. But it also sometimes printed “MQSC found the following response to a previous command on the reply q” in the middle of the result set. Based on the numbers it appears in these cases to have lost some varying number of the non-persistent responses between when the queue filled and enough free space opened up for the Command Server to continue. Since the reply queue is dynamic it should be impossible to find a “response to a previous command” so this suggests that the Command Server has at least some retry logic though it doesn’t appear to be very robust.1 The other behavior I saw was that no indication of a problem occurred other than receipt of an incomplete result set. This suggests that some of the non-persistent responses simply disappeared. The Command Server does place a single message on the Dead Queue when this happens, if a DLQ is defined. My objection to this is that this is a fatal error in MQ’s primary administrative interface and deserves to be logged in a more robust fashion. An error log entry is permanent and is virtually guaranteed to be recorded. On the other hand, a freshly defined QMgr with IBM’s default configuration won’t be able to receive these so this method of exception reporting can hardly be called reliable. In short this is slightly better than trying to put an error message indicating a QFULL onto the queue which is full, but not by any means robust.1 Roger Lacroix of Capitalware did some testing using his MQ Auditor exit and discovered that on hitting QFULL, the Command Server attempts to put an AMQ8101 message onto the reply queue. When this succeeds, the requesting application (runmqsc in my case) has some positive acknowledgement of the error. However since, by definition, the target queue was full last time the Command Server checked, putting an error message there is probably not the ideal strategy. In my case runmqsc did not consistently report the AMQ8101 error which suggests the Command Server experienced a second QFULL when putting the error message, at least some of the time. Extended outage times The fact that the Command Server fails to write an entry to MQ’s error log when aborting processing has its own set of cascading effects. From the perspective of the client application, the primary indicator of a problem is that the Command Server returns a variable result set for things that should be relatively stable such as DIS QCLUSTER(*). If the results are fed into a dashboard, the variability in the result set shows up as large chunks of the MQ estate winking in and out of existence. Worse, the problem may go completely unnoticed for things that tend to be more dynamic such as DIS CHS(*). Since connections can be volatile anyway, a slightly higher degree of volatility could appear normal until and unless someone was looking for a specific connection and saw it appear to wink in and out of existence. Due to the total lack of error log entries on the queue manager side, the MQ Administrator may spend a lot of time focusing on the client application since this is the only component where the variable result set and/or an error shows up. In my case I ended up taking traces at the client first. When I saw it was emptying the queue without apparent error, I then took traces of amqpcsea (the process name of the Command Server). Only after I saw it was getting QFULL did I think to look at MAXDEPTH on the model queue. Widespread impact The reason for this follow-up post is that the last one characterized the issue as a “runmqsc gotcha” which grossly understates the impact. This is in fact a Command Server gotcha which means it impacts anything and everything that relies on the Command Server. In IBM-land that includes runmqsc, of course, as well as MQ Explorer, the REST API, dmpmqcfg, MFT, AMS, MQXR, and probably a few MQ components I’ve overlooked, all of which are in theory fully supported components of MQ that you might expect to be a tad more reliable than this. Then there are all the IBM products built on top of or that bundle MQ which use the Command Server such as IIB, Message Sight, Tivoli CAM and Omegamon agents, InfoSphere Information Server, Rational Integration Tester, and many, many more. Almost all IBM products stacked atop MQ either talk to the command server routinely, or else have setup wizards that do so at build time. Fortunately, something like WebSphere Retail Server is unlikely to issue the kinds of commands that return massive result sets. On the other hand, these are exactly the kind of result sets something like Rational Integration Tester is likely to call for. Obviously, all the 3rd party products that use MQ’s Command Server are also affected. My current client uses Avada’s IR-360 and by playing around with MAXDEPTH on IR-360’s model queue I was able to get the Command Server to fail in exactly the same way that it did for runmqsc. This issue potentially affects all the commercial and freeware MQ Explorer replacements, administrative tooling, developer tooling, testing harnesses, monitors, PCF-based agents, etc. Keep in mind that even though the only error you see will be on the client side, the QFULL occurs on the QMgr side and the Command Server may not be able to place the error message onto the full queue. The client is limited to doing things like re-issuing the request but there is no “resume DIS QCLUSTER(*) from response #3215” type of functionality that might get it past the previous error. This means that even Paul Clarke’s MQMon program can’t stop the Command Server from losing non-persistent responses or aborting with an incomplete result set. (Although I would not be surprised to find that MQMon gave a more meaningful error to the user than does runmqsc.) IBM’s response My PMR proposed three possible actions to address this: Bump the default MAXDEPTH on the model queue. (See below for issues with this approach.) Enhance amqpcsea to produce an error log entry from it’s QFULL abort routine. Add some guidance in the Knowledge Center that might lead the MQ Admin to bump MAXDEPTH without having to trace everything, open a PMR, or undertake other lengthy diagnostics that extend the outage. The Command Server already emits error log entries for other fatal exceptions. It’s the primary administrative interface for an otherwise rock-solid product, one would hope that it would emit an error log entry for any fatal exception. When reporting to any vendor “hey, the primary administrative component of your product is unreliable under these common conditions” the expected response is along the lines of “Wow, how’d we miss that? Thanks for bringing it to our attention. We’ll get right on it.” Unfortunately, the official response to my PMR amounts to “working as designed,” although thankfully the writer didn’t use those exact words. Unofficially, I’ve pled my case to some of the MQ Dev team management and am hoping this is taken up as a usability issue. The notion that the primary administrative component of MQ fails to emit an error log entry during a fatal exception and it is not considered a defect seems like a perverse result and I’m hoping the folks in the lab will see it that way too. For now though it’s up to us to understand this and come up with our own mitigation strategies. Which brings me to… Mitigation strategies Use persistent messages I know you were thinking “just bump MAXDEPTH” but bear with me a moment. One of the behaviors that I saw was an incomplete result set where some of the missing messages were from the middle of the result set and no AMQ8101 message was received. In other words, partial a result set that looks completely normal unless you already know exactly how many responses to expect and realize this isn’t it. There are plenty of conditions that result in loss of non-persistent messages but do not involve QFULL, and any client-based instrumentation or tooling is more likely to encounter these than something running locally and using native MQ shared memory bindings. Similarly, issuing MQSC commands through an intermediate queue manager can drop non-persistent messages if the XMitQ has a low MAXDEPTH and/or the channel uses NPMSPEED(FAST). The last message of the result set will have the “last in group” flag set. Any scenario in which the messages lost do not include the last one therefore show up as a successful call, even though the result set is incomplete. On the other hand, a QFULL reason code in a Dead Letter Header would be almost as useful to the MQ Admin as an error log entry. Possibly more so because the MQ Admin has to know to go looking for an error log entry in the first place. The Command server could return incomplete result sets for years without anyone noticing, whereas messages landing in the DLQ are as visible as emergency flares. Using persistent messages prevents all the routine cases in which MQ silently discards or loses them so there will be evidence in the DLQ or elsewhere of any issues. The down side is that this strategy requires use of permanent dynamic queues and chances are high that many components aren’t coded to issue delete commands against PermDyn queues. That means it’s up to the MQ Admin to prune the unused orphaned reply queues from time to time. Increase MAXDEPTH on the model queue The obvious fix is to increasing MAXDEPTH so the Command Server never hits a QFULL condition, but this has its own issues. For starters, which model queue or queues do you change and by how much? There are a few well-known model queues: SYSTEM.MQEXPLORER.REPLY.MODEL with default MAXDEPTH(5000) SYSTEM.MQSC.REPLY.MODEL with default MAXDEPTH(3000) SYSTEM.DEFAULT.MODEL.QUEUE with default MAXDEPTH(5000) There are also some not-so-obvious model queues. Many of IBM’s layered products define their own model queue. Many 3rd party products do as well. If the strategy is to bump MAXDEPTH on any queue used by the Command Server, the MQ Admin must be able to identify all of them, including the ones that are not among IBM’s defaults. Assuming all the model queues are correctly identified, how high should MAXDEPTH be? At minimum it must be greater then the number of messages in the highest possible result set. At the other extreme, setting MAXDEPTH(999999999) is one way to ensure the program will never get a QFULL. From a security perspective this setting opens up lots of new vectors for resource exhaustion attacks, especially if the queue in question is SYSTEM.DEFAULT.MODEL.QUEUE. The chance of an attacker exploiting MAXDEPTH on model queues is far smaller than the near certainty in a moderately large MQ shop of the Command Server hitting QFULL. On the other hand, the potential impact of a resource exhaustion attack is loss of the entire QMgr when the filesystem fills up. Maybe this is monitored for in the filesystem. Maybe not. Each shop needs to do their own cost/risk/benefit analysis on this one, but all other things being equal, this is one case in which I don’t mind so much setting a really high MAXDEPTH even though I generally advise against this practice for application queues. Use custom model queues For your own code and any vendor code that allows it, moving to dedicated model queues provides a bit of useful granularity. If you are forced, for example, to bump SYSTEM.DEFAULT.MODEL.QUEUE to MAXDEPTH(999999999) but are able to move your applications to a dedicated model queue with a more reasonable MAXDEPTH then there are fewer vectors for that resource exhaustion attack . In the event of an actual attack or, more likely a misbehaving program, it will be easier to identify the source. Use NPMSPEED(NORMAL) In the case of sending commands through an intermediate queue manager, it’s possible to lose non-persistent replies that traverse fast channels. Setting NPMSPEED(NORMAL) eliminates that point of failure but also impacts all other non-persistent message traffic on that channel. Again, this needs a bit of cost/benefit analysis and I can’t make a blanket recommendation Defensive coding My original post describes some defensive coding techniques for scripts that scrape runmqsc output. On the MQ Listserv Tim Zielke describes a solution for applications that work directly with the PCF responses. He proposes “tracking the number of results that come back and then comparing that number to the MQCFH.MsgSeqNumber for the result that has the MQCFH.Control = MQCFC_LAST.”2 MAXMSGL – wait, what? In addition to selecting an unusually small default for MAXDEPTH on the MQSC Reply queue, IBM also set the MAXMSGL on that queue to a very small non-standard value of about 32k. Presumably someone added up the fields for the largest possible PCF response to come up with this number but I haven’t personally double-checked that. I’d like to think that’s how they came up with that number but then again, until recently I also liked to think the Command Server had robust exception handling and would cut an error log entry for any fatal exception. The behavior on hitting MAXMSGL can easily be tested by setting a low MAXMSGL on the model queue and I have left that as an exercise for the reader. If the Command Server cuts an error log entry or FDC when MAXMSGL is exceeded then perhaps we can leave it at 32k, confident that any failures will be reported. On the other hand, I’m sure some readers will want to bump it to 4mb or even 100MB just on principle. I’m not calling anyone out but the name of the person I have in mind rhymes with Bren Glumbaugh. You know who you are. Final thoughts I’ve been using runmqsc for 25 years without incident, but that’s always been the local version. When I noticed the problem I was able to confirm it by running the same commands locally and over client then comparing the results. The local version consistently gave me the complete result set under all test conditions. The conclusion I draw from this is that a vast body of existing scripts and other tools that use the Command Server probably contain implicit assumptions about reliability and lack robust exception handling. Converting them from local to MQ Client-based operation without also adding robust exception handling may result in some level of unreliability, even when using a really large MAXDEPTH. If you are interested in some defensive coding techniques, go read the original post Client-based runmqsc gotcha. There isn’t a good way to detect the loss of responses in the middle of a result set but this will at least pick up any errors that do get reported. As always, any comments or feedback are greatly appreciated. Much of my feedback tends to be over email for reasons of confidentiality but I try to sanitize it and republish when possible so that the knowledge gained can be useful to others. This post contains feedback from Roger Lacroix and two others, all of whom I am deeply grateful to. Thanks! Updates: 20180514 – Added paragraph on messages put to the DLQ by amqpcsea. This was omitted in the original because I wanted to focus on the need for robust error logging and the explanation of how DLQ messages fail did not seem to add constructively. I have in email been advised this was a bit unfair to the Command Server. 20180514 -Added material provided by Tim on the MQ list regarding defensive PCF coding.
Most shops running more than a handful of queue managers end up writing a script or two that sends commands to runmqsc and parses the responses. Until recently the options were to run those scripts locally or to access remote … Continue reading →
Most shops running more than a handful of queue managers end up writing a script or two that sends commands to runmqsc and parses the responses. Until recently the options were to run those scripts locally or to access remote queue managers using the MQ network. Neither of these options is ideal. Use of local scripts requires maintaining a code base distributed across a variety of hosts and often disparate platforms. Sending runmqsc commands across the MQ network solves those problems but requires a security model in which at least some of the adjacent queue managers can administer each other. The new MQ Client capabilities in runmqsc give us the best of both worlds. We can maintain a single set of scripts in a central location and use them to connect directly to each of the queue managers using MQ client. Some functions that require correlation of state across multiple queue managers are actually easier to build using MQ client. There is, however, a gotcha that anyone using client-based runmqsc should be aware of. I’ll explain that here along with some techniques to mitigate the issue and feedback from the PMR I opened. Workflow The steps to complete a client runmqsc call are as follows: The runmqsc component is executed in client mode on the remote node and requests a connection to the target queue manager. The queue manager’s Message Channel Agent process amqrmppa receives the connection… …creates a dynamic reply-to queue based on SYSTEM.MQSC.REPLY.QUEUE… …and places the command on SYSTEM.ADMIN.COMMAND.QUEUE. The Command Server process amqpcsea receives the command from the queue. The Command Server places the responses on the dynamic Reply-To queue. The Channel Agent transmits the messages over the network. The runmqsc client receives and displays the response set. runmqsc–>amqrmppa–>amqpcsea–>amqrmppa–>runmqsc Note that runmqsc connecting to a local queue manager might also use the command server. Although it could directly access MQ objects in this scenario, it would make sense for it to use similar code paths in local and client mode. This is far less complex than having one code path to use local, privileged, internal MQ calls and another completely different code path for clients. For purposes of this discussion the important thing about runmqsc running locally is it talks to MQ directly, which is to say it’s really fast. What’s changed? When IBM delivered runmqsc client capability they also gave us a new model queue named SYSTEM.MQSC.REPLY.QUEUE dedicated for that use. In a curious and typically inscrutable design decision, IBM set the default MAXDEPTH for this queue to 3,000. Since this is less than the system-global default of 5,000 set on most other queues one assumes there is some risk that is mitigated by choosing a more conservative setting. Whatever that risk is must be significant since the trade-off is that client runmqsc behaves unreliably as the number of response messages approaches and exceeds this value. How it breaks A client-based runmqsc creates a fast producer/slow consumer situation because amqpcsea can dump messages into the reply-to queue much faster than amqrmppa can retrieve them and deliver over the network. Accordingly, a runmqsc client call that results in more than MAXDEPTH responses runs a risk of overrunning the queue. Recently I’ve been converting my Cluster Health scripts to use runmqsc client and the cluster I’m working with has more than 4,000 advertised queue objects. Over successive runs the total number of responses returned to runmqsc fell short of the expected total about 60% of the time. This meant that about 40% of the time amqrmppa was able to pull the first 1k messages from the queue before amqpcsea was done writing the full 4k messages so the reply-to queue never filled up. The other 60% of the time were cases in which amqpcsea encountered a 2053 QFULL condition and the exception handling did not recover correctly. I tested this by adjusting MAXDEPTH very low and very high and got back the expected behavior reliably in both configurations. Whenever MAXDEPTH was low the error rate increased. Setting it to a very high value completely eliminated the errors. Why is this a ‘gotcha’? The problem is that neither runmqsc nor amqpcsea provides an error message that might lead the MQ Admin to adjust MAXDEPTH on the model queue. In fact, amqpcsea doesn’t provide any error message at all. Anyone trying to track this down on the queue manager side will find no clues whatsoever. No error log message, no event message, no FDC. On the client side the messages can be buried among the responses and go completely unnoticed. Because the Command Server doesn’t provide header or footer records with totals to reconcile back to, it is not immediately obvious when partial result sets are received. A script that doesn’t explicitly look for these error messages will probably fail silently and use the partial result set as if it were complete. The chance this will cause problems is a very high number that approaches certainty. Over more than 20 years participating on the MQ Listserv, in various other online communities, and attending more conferences than I can count, I don’t remember any discussion on the topic of runmqsc results being unreliable. I point this out to support my next assertion which is that the vast body existing scripts built around runmqsc virtually all contain the implicit assumption that the result sets returned are complete. When using runmqsc client, this level of reliability can no longer be taken for granted. In making this claim I do realize that converting scripts to use client-based calls introduces the entire class of errors related to connection management and that tech-savvy MQ Admins will have already added some exception handling code to their scripts. Great! But what conditions do these scripts look for? Some of the errors I encountered are brand new. I looked through the Knowledge Center to assemble a list of error messages but am to this day not confident I’ve identified the complete set. Even if I have, this is new function and IBM may add a new error message as they refine their internal exception handling. Practical workarounds The first and most obvious workaround is to bump MAXDEPTH on the SYSTEM.MQSC.REPLY.QUEUE to some value much higher than 3,000. This is where I would really like to know IBM’s reasoning for choosing the value they did because the last thing I want to do is get into an iterative cycle of fail-and-fix. My preferred solution would be to set a MAXDEPTH value that I will never need to adjust again, and then make that part of my baseline. Since I can’t imagine any harm in setting it to a high value, at least nothing that is worse than the behavior it exhibits with the default, I set it to 500,000. The other approach I took was to avoid looking for specific error messages. The messages to expect from a good result are well known. When running CLUSQMGR the replies come back as AMQ8441 messages. Similarly, QCLUSTER commands return AMQ8409 messages, QSTATUS commands return AMQ8450, and so on. So rather than look for specific errors, I just look for messages that are not the expected set. For example: CLUSQM="$(echo "DIS CLUSQMGR(*) CLUSTER($CLUSTER) ALL" | runmqsc -e -w 10 -c $QMGR)" ERR=$(echo "$CLUSQM" | grep -v -e '^AMQ8441' | grep -e '^AMQ[:digit:]+' | wc -l) This fetches the results of DIS CLUSQMGR(*) and then counts all the AMQ messages other than the expected ones. If $ERR ends up with a non-zero value we can take appropriate action even if the specific error is unknown when the script is written. If over time unhandled AMQ messages that are routine and expected make themselves known by tripping the error code, we simply extend the grep -v call with another -e pattern. The PMR and IBM’s response Over the years my consulting clients have typically been very large banks, insurers, state and federal government agencies, and other large institutions. Virtually all of these large shops have queue managers and clusters with more than 3,000 objects. Extrapolating from there I’m assuming that virtually all of IBM’s best MQ customers will find client-based runmqsc to be unreliable. Based on this I offered IBM two different proposals. The first was that if runmqsc client is “working as designed” then the Knowledge Center should be updated to explain the issues, what the MQ Admin needs to look for, and the suggested remediation. If there’s a compelling reason for SYSTEM.MQSC.REPLY.QUEUE to have a low MAXDEPTH that can be explained here and users can weigh that against the unreliable behavior. Of course, that’s the worst-case fix since it requires IBM to document that runmqsc is routinely and silently unreliable but that this is by design. My preferred proposal is that IBM would enhance amqpcsea to emit error log entries, and possibly event messages, when encountering a QFULLL condition. This would allow for automated detection of the events as well as leave a bread crumb trail for the MQ Admin. Bumping MAXDEPTH of the model queue would be helpful but does not address root cause unless it gets bumped to 999,999,999 similar to the Dead Queue. IBM has confirmed that all of this revolves around the Command Server hitting QFULL. They have not provided much detail as to the level of exception handling or accounting for the variety of error messages seen on the client side. Their initial response was that they have passed my enhancement request to the lab but that QFULL is a generic condition and not appropriate for error logs or event messages. I replied that the concern is limited to amqpcsea and in that context a QFULL should be a reportable error because it makes the result set unreliable. As an MQ Admin I don’t want QFULL instrumented. I want MQ to be reliable or to alert me to conditions in which it isn’t rather than silently omit half a result set. Next steps Obviously, I will post the final results when the PMR is closed. Check back, subscribe, or watch for an announcement on the Vienna MQ Listserv if you are curious. Regular readers who made it this far in the post are surely thinking “he said ‘MQ client’ but didn’t mention security!” Correct! If you attend this year’s MQTC I’ll address this at one of my sessions which will focus on getting runmqsc and dmpmqcfg to run in client mode over mutually authenticated TLS channels.
Every now and then we hear that a cipher algorithm has fallen to a new cracking technique. This cascades into a new round of deprecating any ciphersuites that rely on the newly cracked algorithms. Over the years we’ve moved from … Continue reading →
<img decoding="async" class="alignnone size-full wp-image-1096" src="https://t-rob.net/wp-content/uploads/2018/01/2018-01-19_14-15-13.jpg" alt="" width="600" height="150" srcset="https://t-rob.net/wp-content/uploads/2018/01/2018-01-19_14-15-13.jpg 600w, https://t-rob.net/wp-content/uploads/2018/01/2018-01-19_14-15-13-150x38.jpg 150w, https://t-rob.net/wp-content/uploads/2018/01/2018-01-19_14-15-13-300x75.jpg 300w" sizes="(max-width: 600px) 100vw, 600px" /> Every now and then we hear that a cipher algorithm has fallen to a new cracking technique. This cascades into a new round of deprecating any ciphersuites that rely on the newly cracked algorithms. Over the years we’ve moved from SSL to TLS, from DES to 3DES, from MD5 to SHA, and so on. The list of items on MQ’s Deprecated Ciphers page grows slowly but relentlessly over time. But not all of the ciphersuites that IBM has deprecated are broken. So why are they on the list and should we use them? This post will attempt to shed some light on those questions. Non-deprecated deprecation Sprinkled among all the other deprecated ciphers are a few with no deprecated component As of this writing, these include: ECDHE_ECDSA_NULL_SHA256 ECDHE_RSA_NULL_SHA256 TLS_RSA_WITH_NULL_NULL TLS_RSA_WITH_NULL_SHA256 Note that all of the component protocols—ECDHE, ECDSA, TLS, RSA, and SHA256—are present in one or more ciphersuites that are not deprecated. On the other hand, none of the non-deprecated ciphersuites contain a NULL component. The obvious conclusion is that these entries made the list because of the NULL components of their respective ciphersuites. NULL hasn’t been cracked because there’s nothing to crack. So these aren’t deprecated because of a crypto weakness but rather because of a policy that discourages the use of NULL ciphers in a ciphersuite. Is that really a good thing? The case for null ciphers TLS provides three different security functions: authentication, privacy (encryption), and integrity (hashing). The ciphersuite name is the combination of the ciphers used for each function, the last two components being the type of encryption and the type of integrity check. So TLS_RSA_WITH_NULL_SHA256 indicates a ciphersuite in which… The TLS protocol uses RSA for authentication. No encryption is used therefore no privacy is provided. Integrity checks use SHA256 hashing. Switching from NULL_SHA256 to NULL_NULL results in a ciphersuite that provides authentication but no encryption and no integrity checking. Consider a new requirement that specifies authentication of existing plaintext MQ channels. The administrator can use the TLS_RSA_WITH_NULL_NULL ciphersuite to provide that authentication without the overhead of adding encryption and hashing of all the data which, although that isn’t a bad thing, was not given as a requirement. In some cases where performance is critical, a requirement is stated specifically to omit crypto on the data stream. Deprecating ciphersuites with NULL components removes the administrator’s ability to specify that a channel will be authenticated without also using integrity and/or privacy on the data. There are plenty of cases where channel authenticity is required but integrity and/or privacy is not. MQ used to recognize this back in the days when we had NULL_MD5 and NULL_SHA, so what happened? Deprecation from a security perspective When the crypto community deprecates algorithms like MD5 or SHA1, it has nothing to do with the strength of the cipher. It’s more about the delta between achieved vs. perceived strength, and that this gap expands dramatically after the algorithm is broken. Prior to being cracked these ciphers were strong enough for some intended use but now, assuming we know they are broken, we should consider them as providing zero protection. From a more pragmatic standpoint, the usual method of choosing cipherspecs is to pick the ones that provide the best performance for a given level of protection. For use cases in which zero protection is required for privacy or integrity, the NULL cipher is a lot more performant than any broken cipher. Why would anyone deliberately choose the overhead of DES, for example, when NULL meets the requirement? From that perspective, ciphers with SHA1 or MD5 provide a mere façade of security. People who aren’t aware they are broken might have a false sense of security seeing some non-NULL cipher specified. However, there is no mistaking a NULL cipher as providing any security. The delta between the security perceived and that achieved is zero – exactly what we want. A realistic sense of insecurity is preferable to a false sense of security every time. Are these deprecated by mistake? <img decoding="async" class="alignright wp-image-1097 size-thumbnail" src="https://t-rob.net/wp-content/uploads/2018/01/Fotolia_50112717_S-150x150.jpg" alt="" width="150" height="150" srcset="https://t-rob.net/wp-content/uploads/2018/01/Fotolia_50112717_S-150x150.jpg 150w, https://t-rob.net/wp-content/uploads/2018/01/Fotolia_50112717_S-300x300.jpg 300w, https://t-rob.net/wp-content/uploads/2018/01/Fotolia_50112717_S.jpg 693w" sizes="(max-width: 150px) 100vw, 150px" />There are valid use cases for ciphersuites with NULL components and the non-NULL ciphers used in these ciphersuites are still considered strong. It’s easy to look at this and wonder if these ciphersuites were put on the list by mistake. I do not believe that to be the case. The IBM MQ team clearly knows about use cases for NULL privacy and/or integrity. We know this because in AMS these are considered features. AMS has always had Protection Policy of Sealed–messages that are signed but not encrypted and thus provide authentication and integrity. The recent addition of Confidentiality policy–privacy and integrity functions but no authentication–was significant enough to be included in conference presentations and press releases. Clearly, the MQ Dev team knows about use cases that do not call for authentication, privacy, integrity, or some combination of two of these. Why deprecate them, then? So why did IBM deprecate these ciphersuites in MQ? Short answer–because everyone else does. No, really. Let me explain. OWASP, the PCI Security Council, and many other industry groups whose primary concerns are Internet-facing web servers do not rely on TLS for authentication at all, but absolutely require privacy and integrity at all times. In a purely web context, this policy makes absolute sense. But because OWASP and other industry groups deprecate weak and NULL ciphers for web servers, lots of scanners flag them. Point one of those scanners at a queue manager and any ciphersuite with a NULL cipher is flagged. If these were not deprecated, lots of people would be opening PMRs with subject lines like “Help! The security scans of MQ are flagged for weak ciphers!” That would then require some mechanism to disable all the ciphers with a NULL component. Which brings us to back to the current approach. IBM deprecates these by default but provides the means for the MQ administrator to selectively enable one or more of them. By making the enabling of ciphers a deliberate administrative choice, IBM and the MQ Administrator can be confident that the reason the QMgr gets flagged by the security scanner will be obvious. (In some cases that confidence may be misplaced but those are hopefully the exception.) The solution will perhaps not be so obvious since that depends a lot on local policies and exception procedures, but at least the MQ parts of that solution will be known to the administrator. She or he can always back out the configuration that enabled the flagged cipher in the first place if that’s preferable to running the gauntlet of requirements to obtain scanner exceptions. Not all ciphersuites with NULL are equal The only down side to this arrangement is that the Deprecated Ciphers page doesn’t distinguish between ciphersuites that are listed because they are broken versus those on the list because they contain NULL ciphers. These two sets currently overlap at the NULL_SHA and NULL ciphersuites. These are older ciphersuites that use the broken SSL protocol for authentication. The use case may allow NULL ciphers but the crypto it does call for, authentication in this case, needs to be strong. One of the TLS or ECDHE ciphers with NULL components is preferable to NULL_SHA or NULL_MD5 in this case because the authentication portion must be reliable. Over time the intersection of these two groups will grow larger. RSA is already waning in favor of protocols like EDCHE that use ephemeral keys and provide perfect forward secrecy. Extrapolating forward, we can predict a time when TLS_RSA_WITH_NULL_SHA256 will be as undesirable as NULL_SHA is today. When that day comes the MQ administrator will need the skill and knowledge to distinguish good from bad ciphersuites. The key, if you will excuse the pun, will be for the MQ administrator to understand which of the ciphers are on the deprecated list because they were broken and which are on the list purely for policy reasons. Putting it all together The addition of CONNAUTH to MQ has been driving a wave of reconsideration of authentication methods in MQ. Certificates are often preferable to passwords for headless application servers and service accounts, but at times not when doing so incurs the overhead of crypto on all the data traffic. The use of a ciphersuite that authenticates the connection but doesn’t encrypt and/or hash the traffic can resolve that issue. Where the use case requires it, don’t be afraid to to enable one of the appropriate deprecated ciphersuites. Use the syntax that enables these selectively rather than enabling all of them, picking only the one that is needed. Be prepared to respond to the security team who run the internal scanners, though. There is a good chance that enabling one of these ciphers will trigger a flag on the scans that will require an exception. Looking longer term, Elliptic Curve crypto may resolve the tension between performance versus scanner flagging. EC is more performant for a given strength than older ciphers so it might be possible that the hashing and encryption do not break the performance bounds required for the channel. The difficulty in switching to Elliptic Curve is that personal certificates for EC are incompatible with older ciphersuites so migrating may require new certs for MQ and the application. MQ’s ability to use more than one personal certificates makes it possible for a single queue manager to support older and newer EC-based ciphersuites at the same time, thus providing a path for gradual migration of one application at a time. What direction will you take? <img loading="lazy" decoding="async" class="alignleft size-thumbnail wp-image-1098" src="https://t-rob.net/wp-content/uploads/2018/01/Fotolia_24840613_S_RT-150x54.jpg" alt="" width="150" height="54" srcset="https://t-rob.net/wp-content/uploads/2018/01/Fotolia_24840613_S_RT-150x54.jpg 150w, https://t-rob.net/wp-content/uploads/2018/01/Fotolia_24840613_S_RT-300x109.jpg 300w, https://t-rob.net/wp-content/uploads/2018/01/Fotolia_24840613_S_RT-768x278.jpg 768w, https://t-rob.net/wp-content/uploads/2018/01/Fotolia_24840613_S_RT-1024x371.jpg 1024w, https://t-rob.net/wp-content/uploads/2018/01/Fotolia_24840613_S_RT.jpg 1151w" sizes="auto, (max-width: 150px) 100vw, 150px" />However we approach it, some sort of authentication is required for connections to MQ and chances are CONNAUTH or certificates play a part of that for any given queue manager. Just don’t let the deprecation of ciphersuites will NULL components keep you from using certificates in cases where performance is critical, once you know why and how to use them appropriately. As always, feel free to contact me with questions or comments. These articles always generate more emails than posted comments, due to the sensitive nature of the topic and that’s okay. I try to answer all emails and will always hold them in strict confidence, although you might recognize your anonymized use case in a future blog post or conference presentation. Do post comments though when possible so that others benefit from the discussion.
A how-to tutorial and code samples for parsing #IBMMQ error logs in @Splunk. #IBM @IBMChamp Continue reading →
A growing number of my clients are deploying IBM MQ on Amazon EC2 instances and a common need that I see emerging is for instrumentation and tooling. When the MQ instance is ephemeral, deploying instances on demand and decommissioning just as suddenly, lots of things the MQ Admin used to do by hand need to be automated. This includes build-time things such as defining objects, run-time tasks like enabling or disabling queues in the cluster, and forensic capabilities such as archiving error logs. It is this last item that concerned a recent customer. Their main requirement was to ingest MQ error logs in real time, or at least close to it, so those logs would survive death of the virtual host on which they were generated. Getting Splunk to ingest the logs was ridiculously easy. Just define the log files as a Splunk data input and immediately they become available through the Splunk search interface. That’s all well and good if all you want to do is browse through the logs or search them for particular error codes. To get the benefit of Splunk analytics requires the error logs to be parsed into fields. Then instead of merely searching for error codes you already know about, you can ask Splunk to show you a report of all the error codes sorted by prevalence, by frequency over time, or even which ones are the rare outliers. All the analytic capabilities are usable once the fields are parsed. Better yet, parse logs from many queue managers and now you can spot trends or pick out nodes that are showing early signs of distress. That’s really useful stuff and Splunk provides it right out of the box, but only for log types it knows how to parse. So let’s teach it to parse IBM MQ error logs, shall we? MQ error log structure Before wading into the transform code, let’s look at a couple of typical IBM MQ error log entries. The first of these is reporting an error. The second one reports a normal channel startup. 12/18/2017 04:12:54 PM - Process(99475.1) User(mqm) Program(runmqchl) Host(mq8007.ioptconsulting.com) Installation(Installation1) VRMF(8.0.0.7) QMgr(ASH) AMQ9999: Channel 'ASH.BIRCH' to host 'localhost(11412)' ended abnormally. EXPLANATION: The channel program running under process ID 99475 for channel 'ASH.BIRCH' ended abnormally. The host name is 'localhost(11412)'; in some cases the host name cannot be determined and so is shown as '????'. ACTION: Look at previous error messages for the channel program in the error logs to determine the cause of the failure. Note that this message can be excluded completely or suppressed by tuning the "ExcludeMessage" or "SuppressMessage" attributes under the "QMErrorLog" stanza in qm.ini. Further information can be found in the System Administration Guide. ----- amqrccca.c : 1090 ------------------------------------------------------- 12/18/2017 04:12:54 PM - Process(100143.1) User(mqm) Program(runmqchl) Host(mq8007.ioptconsulting.com) Installation(Installation1) VRMF(8.0.0.7) QMgr(ASH) AMQ9002: Channel 'ASH.BIRCH' is starting. EXPLANATION: Channel 'ASH.BIRCH' is starting. ACTION: None. ------------------------------------------------------------------------------- Splunk is really good at finding timestamps so without any customization it splits the multi-line log events as you might expect, based on each event beginning with a timestamp and only one timestamp per event. After that it gets a bit more challenging. Splunk will tease out fields when it finds them in the format key=value but as seen here, MQ doesn’t use that format. At the top are some fields in the format Key(Value). These are followed by the error code and text fields. These are represented as bare values so we will need to provide the key names. After that come the explanation and action fields, and these are delimited by a colon and a newline and contain variable text which may extend over one or more lines. Notice also that the eyecatcher between entries, the line with all the dash characters, has optional fields that contain the name of the C source file and line number within that. These are not always present but when they are there they are worth capturing. The example error entries above are from the ASH queue manager in my MQ sandbox. MQ also maintains a set of server-global error logs for use when the error is not specific to a queue manager or the queue manager can’t be determined. Ideally, we want to parse MQ’s server-global error logs along with the queue manager error logs, but entries in the global logs will not have a field containing the queue manager name. That means the queue manager name must be an optional field, similar to the C source and line number fields. Here is an example of a global error log entry that lacks the QMgr field: 12/18/2017 05:17:50 PM - Process(105765.1) User(peermap) Program(setmqinst) Host(mq8007.ioptconsulting.com) Installation(Installation1) VRMF(8.0.0.7) AMQ8584: Insufficient permission to update installation configuration. EXPLANATION: An attempt was made to update the WebSphere MQ installation configuration for installation 'Installation1' path '/opt/mqm' but the request was rejected as the current user does not have sufficient authority to make the update. ACTION: Issue the command from a user with sufficient authority to update the installation configuration. ----- amqiset0.c : 922 -------------------------------------------------------- Requirements for a transform Splunk needs us to tell it a few basic things about the structure of the logs: Since this is a multi-line log format, Splunk needs to know how to tell when one entry ends and a new one begins. Delimiters for any key/value pairs in a log entry. This includes the delimiters between pairs, as well as the delimiter that joins the key to the value in the pair. How to find any fields that are not in a key/value format. Names for any fields that are not in a key/value format. Which fields, if any, are optional. All of these things must be expressed in regex terms. Although Splunk allows for the definition of regex primitives that are combined to make more complex constructs, I chose to build one big regex. It’s easier for me to work with and test that way, and it meant I didn’t need to learn how to define and reference the primitives. I suspect this is one area where a more experienced Splunk admin would take a different approach. If so, I’d love to hear about it in the comments. Entry boundaries Splunk is really good at finding and parsing timestamps so we don’t need to give it any help there. But we do need to tell it how to find the first field after the timestamp and how to find the end of the entry. This regex identifies the beginning of the record: ^[^-]+- Process From the beginning of the line, look for one or more characters that are not a dash, followed by a dash, a space and the word ‘Process’. The end of the record is a bit simpler: -+\s*$ That’s one or more dash characters, zero or more space characters to make the regex a bit forgiving of variations in line endings, all leading up to the end of the line on which these appear. Key/Value pairs Although it’s possible to specify delimiters and let Splunk parse the field names from the data, I wanted to define an IBM MQ namespace so I assigned my own field names. The first two fields are parsed by the following regex: Process\((?P<ibmmq_err_process>[^\)]+)\)\s+User\((?P<ibmmq_err_user>\w+)\)\s+ This finds each field by its literal name, then extracts the value between the parentheses and assigns it to the field name in angle brackets. The ?P is Splunk’s notation for a named field. The field name appears within angle brackets and the value is expressed as [^)]+)\) which means “one or more characters up to but not including a right parentheses.” The unescaped right parentheses is where the field-capture ends and the escaped one is the literal we see in the log entry. Between the two fields is \s+ which means “at least one and possibly more whitespace characters.” Based on the log sample we have today a simple literal space would work. Allowing for variable number of whitespace characters makes the regex tolerant of some variation there such as future use of tabs or spaces by IBM to align fields. Bare fields The next two fields are the error code and the error text. Strictly speaking this is a key-value pair where the error code is the key. But if we let Splunk parse it that way we lose the ability to have Splunk calculate the distribution of error codes over time. What we really want is a single field with the error code itself as the value. The error text can have substitutable values so that isn’t a fixed string even though the error code is. I chose to capture the error text as its own field. Even though the error text was a single line in all my examples, we can’t count on this always being true. Something like a long hostname inserted into the error text might be sufficient to cause the line to wrap and we need to account for that. The resulting regex looks like this: ^(?P<ibmmq_err_code>AMQ[^ ]+):\s+(?P<ibmmq_err_text>.*?)\s+^EXPLANATION: This looks for the literal AMQ at the beginning of the line followed by one or more non-space characters, a space, and a colon, then captures everything from the AMQ to the last non-space character as the error code. The error code in all of the examples I looked at was AMQ and 4 digits but I allowed for characters other than digits and for more or less than 4 of them. This regex does require the error code to start with AMQ and I’m unsure if that holds true for all MQ errors. I suspect that it does not but have yet to see a counter example. For now, this will have to do. The error text was a bit trickier since it can span a variable number of lines. The way I handled this was to gather everything after the colon and space up to the last non-space character before the EXPLANATION: tag. The field captured will thus span any number of lines but doesn’t pick up stray whitespace at the end. Note that the field names are assigned here the same way they were for the key-value pairs. The only difference is that we use the position relative to other fields instead of literal strings to identify the field, then assign names of our own choosing. Line-delimited key-value pairs The Explanation and Action fields are key-value pairs but with different delimiters than before. As you might expect, we capture them in much the same way as before: +^EXPLANATION:\s+(?P<ibmmq_err_explanation>.*?)\s*^ACTION:\s+(?P<ibmmq_err_action>.*)\s*^----+ In both cases we look for the literal string of the field name, followed by at least one whitespace character. Then all the text is captured up to the last non-whitespace character before the next field delimiter. The literal ACTION: at the beginning of a line delimits the terminates the Explanation field. A string of at least five dashes at the beginning of a line marks the end of the Action field. Optional queue manager name Three of the fields need to be defined as optional. The regex syntax allows for an optional group but care must be taken that when the group is absent, the remainder of the regex does not break. This can be seen in the boundaries of the field as represented in the regex below: \s+(?:QMgr\((?P<ibmmq_err_qmgr>[^)]+)\))?\s* When it is present, the queue manager field is bounded by a space character followed by the literal ‘QMgr’. That space will always be there and when the field is present, at least one more will be present at he end as well. But we cannot count on there being two space characters here when the queue manager field is absent so the terminator uses an asterisk instead of a plus sign. The result reads like “a space followed by the QMgr tag and field, if present, followed by zero or more whitespace characters.” Thus we require at least one whitespace character after the prior field, and it might be a newline. When the queue manager name is present, it is recognized by the tag ‘QMgr’ with any number of characters prior to the next right parentheses. We don’t need to worry about whether those are valid characters for queue manager names since we know a right parentheses bounds the field and the name can’t contain one of those. The whole group is bounded by (?: which begins a non-capturing regex group, and )? Which closes the group and uses a question mark to specify zero or one occurrences. Because the whole thing is contained in a \s+…\s* pair, removing the field matches a single whitespace character which is the minimal delimiter that we need to omit the field without breaking the rest of the regex. Optional C source file and line The C source code file name and line number that threw the error are optionally embedded into the terminating eye-catcher line at the end of the log entry. When these fields are present there are always five dashes before the first field, and these start at the beginning of the line. There are also always some number of dashes at the end of the line, varying as needed to left-justify the line. If we wanted to match the eye-catcher without the optional fields, the following regex isn’t exactly pretty but would work: ^----+-+$ This says “From the beginning of a line, match four literal dashes, then one or more dashes, then one or more dashes up to the end of the line.” We know there are at least 5 dashes in all lines, regardless of whether the field is present or not. The four literal and one or more variable dashes matches that. The next iteration of one or more dashes matches whatever remains in the case that the fields are present and also in the case that hey are not. All that is needed now is to add the fields in an optional group: ^----+(?: (?P<ibmmq_err_csource>\w+\.c) : (?P<ibmmq_err_lineno>\d+) )?-+$ Note that the spaces which delimit the fields must be contained within the optional group or it won’t work. The source file name us located by looking for a string of one or more alpha-numeric characters ending with the .c suffix. The line number is at least pretty simple. It looks for one or more consecutive digit characters bounded by space characters. There are a few assumptions embedded here that may break, such as that the source code file will always be written in c and that any insert in this line will look like a file name and line number. If these assumptions do not always hold true then these fields won’t always parse. The good news is that when I intentionally cause the regex to break on these fields all the prior fields still parsed accurately. I suspect this is because these are the last fields in the regex and the line-breaker function resets at the next event boundary. At some point I plan to test how forgiving Splunk is if I break the regex somewhere further upstream but for now this seems to work. The transform specification You may have noticed the field names I used all began with ibmmq-err-. My intention is to create a field namespace based on IBMMQ. This accommodates additional future formats such as exit log output or FDC files. In keeping with this naming convention, the transform is called ibmmq-err to match the prefix of all the fields it defines. The Splunk docs for transforms.conf suggest a comment describing the transform stanza and the fields it parses. My stanza with the complete regex is below: [ibmmq-err] # Extracts fields from IBM MQ error logs # Extracts: ibmmq_err_process, ibmmq_err_user, ibmmq_err_program, ibmmq_err_host, ibmmq_err_installation, ibmmq_err_vrmf, # ibmmq_err_qmgr, ibmmq_err_code, ibmmq_err_text, ibmmq_err_explanation, ibmmq_err_action, ibmmq_err_csource, ibmmq_err_lineno REGEX = ^[^-]+- Process\((?P<ibmmq_err_process>[^\)]+)\)\s+User\((?P<ibmmq_err_user>[^\)]+)\)\s+Program\((?P<ibmmq_err_program>[^\)]+)[^\)\n]*\)\s+Host\((?P<ibmmq_err_host>[^\)]+)\)\s+Installation\((?P<ibmmq_err_installation>[^\)]+)\)\s+VRMF\((?P<ibmmq_err_vrmf>[^\)]+)\)\s+(?:QMgr\((?P<ibmmq_err_qmgr>[^\)]+)\))?\s*^(?P<ibmmq_err_code>AMQ\d+):\s+(?P<ibmmq_err_text>.*?)\s+^EXPLANATION:\s+(?P<ibmmq_err_explanation>.*?)\s*^ACTION:\s+(?P<ibmmq_err_action>.*)\s*^----+(?: (?P<ibmmq_err_csource>amq\w+\.c) : (?P<ibmmq_err_lineno>\d+) )?-+$ The regex in the stanza is one really long line that wraps. The properties specification The transform is useless without something that references it. A stanza in props.conf performs that purpose by defining a new source type called ibmmq-error-log that uses the new transform. The stanza I have been using is provided below: [ibmmq_error_log] pulldown_type = true REPORT-access = ibmmq-err SHOULD_LINEMERGE = True TIME_PREFIX = ^ category = IBMMQ LINE_BREAKER = ----------([\r\n])+ description = Parse IBM MQ error logs In addition to parsing error logs, I created another transform to parse the output of API exit logs. That transform is specific to a 3rd party vendor program so I did not include it here, but I mention it to explain the use of the IBMMQ category in the props.conf stanza. When selecting source types and in certain displays, all my MQ-related configurations now show up under the IBMMQ category. If you have only the error log transform this adds an extra layer to drill down through and you may wish to omit the category. Most of this is copied from stanzas I found in the default props.conf file. Although this works, it is the part I’m most unsure of since I’m so new to Splunk. The more I read and use the tool the more nuances I discover which lead me to believe there’s lots of room for improvement here. For example, I read some docs that suggest SHOULD_LINEMERGE can be set to false since a line breaker is specified. I have yet to try this but I plan to set up a virtual machine with checkpoints so I can do A/B testing to figure out how all these stanza elements work. Configuring data inputs Once the stanzas are entered into the local props.conf and transforms.conf files and Splunk restarted to pick them up, all that is left is to set up the data inputs. Splunk is good enough to allow wildcards in path names to I used /var/mqm/errors/AMQERR01.LOG and /var/mqm/qmgrs/*/errors/AMQERR01.log and set them to the new source type of ibmmq-error-logs. The first path grabs the global error log files and the second one gets the error logs of any queue managers defined today or in the future. I like configurations that are as future-proof as possible. These are the default paths and the queue manager’s logs or the global log might be located elsewhere. A possible future enhancement would be to use MQ’s own tooling to tell Splunk where to find the logs for a given queue manager. The dspmqfls command will parse the mqs.ini and the qm.ini and tell you exactly where a queue manager lives. You might have noticed that I specified only the AMQERR01.LOG file in each case. While it is true that there are three files, all of the info in the 02 and 03 files was run through the 01 file first. What I do not want is to get two or three entries for each log event by having Splunk parse all three logs. By using only the AMQERR01.LOG file there is a slight window in which MQ can write to the log while Splunk is not looking then rotate the logs and a few log entries are lost. That window can be reduced by bumping the size of the error log up from the default so that the files rotate less frequently. This was the option my client chose and why my data inputs capture only the 01 file. At some point I’ll test using all three files and force some log rollover events to see how they are indexed but that’s a future enhancement. Splunk MQ error source type in action <img loading="lazy" decoding="async" class="aligncenter size-large wp-image-1087" src="https://t-rob.net/wp-content/uploads/2017/12/2017-12-18_21-16-06-1024x419.jpg" alt="" width="640" height="262" srcset="https://t-rob.net/wp-content/uploads/2017/12/2017-12-18_21-16-06-1024x419.jpg 1024w, https://t-rob.net/wp-content/uploads/2017/12/2017-12-18_21-16-06-150x61.jpg 150w, https://t-rob.net/wp-content/uploads/2017/12/2017-12-18_21-16-06-300x123.jpg 300w, https://t-rob.net/wp-content/uploads/2017/12/2017-12-18_21-16-06-768x314.jpg 768w, https://t-rob.net/wp-content/uploads/2017/12/2017-12-18_21-16-06.jpg 1110w" sizes="auto, (max-width: 640px) 100vw, 640px" />As you can see from the screen shot, the fields defined in the transform are all available for selection on the left and visible as parsed fields in the log record entry itself. Clicking a field raises the usual Top 10 dialog and options to analyze that field over time. Pretty much anything you can do with Splunk analytics is now available for the IBM MQ error logs. Closing thoughts This was tested with MQ v8.0 and Splunk v7.1. There are many variations in the MQ error logs over the years but the basic format is pretty stable. Supporting minor variations in the key-value pairs should be doable by making some fields optional or adding new optional fields. I will update on GitHub with any changes as I am able to test with more versions of MQ and/or Splunk. Feel free to send a pull request, although you may need to ping me by phone or email if I miss the Splunk notification. I’m not on there every day. There are undoubtedly many ways to do this within the Splunk framework and I’m not suggesting this is the best way, or even that this is the “right” way according to Splunk best practices. This is merely what worked for me and I’m posting in the hope that others find it useful. Any improvements or suggestions are welcome. You can find the source code for this on GitHub at /tdotrob/IBMMQ-Logs-Splunk.
After I tweeted a link to an IBM blog post on how to start and stop IBM MQ using systemd, an IBMer responded to say “it surprised me to hear that some #IBMMQ customers have to manually restart their QMs … Continue reading →
After I tweeted a link to an IBM blog post on how to start and stop IBM MQ using systemd, an IBMer responded to say “it surprised me to hear that some #IBMMQ customers have to manually restart their QMs when the box comes up.” My reply was brutally frank: “Should be no surprise – the serviceability gap with MQ start/stop API resembles an open-pit mine. As a result most shops either don’t do it well, or don’t do it at all. Mortgage payments on the technical debt needed here desperately.” Not wanting to leave that hanging out there with no explanation, this post describes in excruciating detail what’s wrong. Hopefully, that’s the first step to getting it fixed. So you want to start MQ? Sounds easy enough, right? Just write a systemd service to kick off strmqm and move on, yes? Not so fast! Unless there’s a default queue manager, the service has to at least know the name of the queue manager to start and if there is more than one queue manager, the service needs to know about all of them. The IBM blog post addresses this by using systemd templates. There’s one generic file with the ability to pass in a substitution parameter. Each queue manager gets a symbolic link to the template file. The link file name contains the queue manager name which is the source of the substitution. So, assuming we have a good template and several links we are good to go, yes? Not so fast! Remember that, like any other boot-level service, systemd runs as root and its configuration files must therefore not be writable by anyone who isn’t root. That means if the recommend practice of giving every queue manager a unique name is followed, the MQ administrator must engage the UNIX/Linux System Administration team for every invocation of crtmqm or dltmqm. So you want to start multi-installation MQ? Multiple MQ installations on the server breaks the assumption that starting a queue manager is as simple as running strmqm. Now the systemd service is obliged to properly set the environment for each of the queue managers it needs to start. Since systemd cannot source setmqenv, the MQ admin must either write an MQ start script to perform this function or else use crtmqenv to generate an environment file for systemd to reference. There are issues with either of these solutions. Using a shell script to set the environment and kick off MQ makes it much harder for systemd to determine which process represents the queue manager’s service. It is customary for daemons to either execute and stay active, or else use one level of fork to run the daemon in the background. MQ natively forks strmqm to start the Execution Controller which becomes the parent of all the other processes. Insert a shell script in there and now the Execution Controller is three processes deep. There is no PID file so systemd just takes a best guess as to which process represents the health of the service. The high cost of root The alternative to creating an environment file and embedding it into the service may be worse. In addition to now requiring root for every invocation of setmqm, this approach suffers from the more general problem of increasing the amount of customization embedded in the systemd configuration files. Shops that value security impose a fairly high cost to obtain root privileges, specifically to minimize the amount of routine work performed as root. Passing that cost on to the requestor in the form of elevated approval, formal change control, request latency, and other means, creates a strong incentive not to request root in the first place. As a result, the more customization is required in the systemd configuration files, the less dynamic the MQ network becomes. Once something is set up, it is set in stone. Some shops have a system of streamlined privilege delegation such that the MQ Admin can obtain root and perform maintenance on the systemd files directly. While this reduces the cost to the MQ admin team, some of that cost is transferred to whoever manages privilege escalation. If it’s managed using a commercial product, the company has the cost to buy it. In addition to those costs, the burden of audit proof rises with the volume of privileged access logs because there is more log content to wade through. Meanwhile, the ability to spot anomalous activity in those same logs decreases as they grow larger. Anything that requires root for mundane daily tasks or to store tunable per-instance configuration imposes very high barriers to adoption. As long as that day-to-day ongoing cost exceeds the cost of occasional downtime to manually start MQ after a server reboots, MQ won’t ever be enabled under systemd. So you want to start multi-instance MQ? First question: how can systemd tell if a given queue manager is multi-instance? It’s a serious question. There is no permanent configuration setting to query and no command one can issue to determine before starting it whether a given queue manager is intended to run as a singleton or as multi-instance. The way that systemd can reliably know a queue manager is supposed to be multi-instance is that someone obtained root privileges and hard-coded strmqm -x into the service template. But wait…doesn’t that mean all queue managers on the box must be multi-instance or else all of them must be singletons? Sure, if there’s only one template. If you want to mix singleton and multi-instance queue managers on the same box you need to create and maintain template files for each type, or else get creative (read: complex) with pre-start capabilities of systemd. At the end of the day what his means is that even more routine root-level access is required in order to maintain the growing set of custom per-instance configuration embedded into systemd files. So you want to stop MQ? If you got this far, we can assume your service files are both multi-installation and multi-instance aware with regard to starting queue managers. The next question is “What command do you use to stop them?” Ideally the queue manager shutdown should proceed through attempts at controlled shutdown, then immediate shutdown, then preemptive shutdown, then finally manual shutdown. Unless you opt for a one-size-fits-all solution that doesn’t shut down in stages, there is no getting around the need to run a shutdown script. A shutdown script isn’t optimal, but at least it won’t confuse systemd‘s PID tracking like start scripts can. What’s in the script? The last one of these I wrote worked like this: Set a timer and issue endmqm -c If the timer pops and the queue manager has processes left, repeat using -i, then -p. If any processes remain, kill them in the order prescribed by IBM. Note that the processes and recommended order change by version so it is necessary to query the queue manager’s version and dynamically construct the kill list from a version table. Shut down any satellite processes that were not started as child processes of the queue manager. Since MQ services always run as mqm, anything that has to run with non-administrative privileges will likely have been started some other way and may hold shared memory segments until they are killed. If there are multiple queue managers, all of this should be happening in parallel if we hope to get everything shut down before the system reaper shows up and starts wantonly killing random processes. Robustly managing multiple parallel child processes from a bash script is not a trivial requirement. So you want to stop multi-instance MQ? This is where things get a bit complicated. I know you are probably saying “Wait, what? Wasn’t all that other stuff complicated?” Sorry, no. Not by comparison. The multi-instance queue manager is a single logical construct composed of processes running in two physical places. Up to now we’ve dealt only with the behavior of those local processes. We hard-coded whether a queue manager was intended to run multi-instance by adding -x to the strmqm command if it was. Now we actually need to know the current and the desired state of the logical queue manager before telling the local processes what to do. For example, let’s say you started the queue manager with -x. Intuitively, you should be able to stop it with -x too, right? Nope. That only works if it is running as the standby instance. If it is the active primary instance you need to use the -s command because the -x is ignored. Well, sort of. The -s parameter tells the queue manager to switch to the standby node. If the standby instance is not running when endmqm -s is issued, it will not be possible to switch instances. In that case the queue manager stays running and emits the error AMQ7276: WebSphere MQ queue manager cannot switch over. What you really need to do in this case is issue endmqm without the -x and without the -s. In short, stopping a multi-instance queue manager requires the stop script to first query the state of the logical queue manager, then issue endmqm with either the -x, or the -s, or neither, depending on the current state of the queue manager and the intended state after running the command (i.e. queue manager remains available on another node or shuts down on all nodes). To make sure the MQ admins are never bored, the tools that query the state of a multi-instance queue manager don’t tell whether the queue manager is intended to be a singleton or a multi-instance queue manager, only which mode the running instance was started in, and nothing at all about HA if at least one running instance isn’t visible to the dspmq command. Just for kicks, IBM threw a -r in there to tell clients whether to try to reconnect or not, and all of the -x, -s, and -r options interact with the -c, -i and -p options. For the sake of brevity I won’t go into detail about the interactions between all of these other than to say that shutting down a multi-instance queue manager sets a high skill bar that every customer must clear. Interactively we can eventually get MQ shut down using trial and error. That’s the lowest common denominator across all 10,000 or so MQ shops. It is what we do by default. Coding it is a different story. An admin can manually execute thousands of MQ start or stop events without exceeding the resource cost of correctly implementing all the exception handling needed by a shutdown script. Based on that, you should be able to correctly guess what approach many of us take and why. So you want to start or stop MQ from the command line? You really are a glutton for punishment, aren’t you? When systemd starts processes, it sets their resource limits, their kernel tuning, their environment, and more. It also places every process that is spawned in a control group named after its service. Start MQ as a service, it gets the environment and tuning specified for the MQ service. Start MQ from the command line and MQ gets the environment configuration tailored for the getty service that manages user login sessions. Start several queue managers from the command line, they all share the pool of resources allocated to that user by the getty service. Whoops. The variety of possible outcomes there can make for some interesting troubleshooting. What’s an admin to do? The standard advice is to use systemctl to start and stop MQ services. In case you are not familiar with systemctl, running it is one more task that requires root. The MQ shop is faced with the choice either to escalate to root for every invocation of strmqm and every invocation of endmqm, or to try to reconcile the environment of the getty service to the environment of the mqm service and hope for the best. Or, perhaps avoid using systemd to start MQ. A more subtle effect of this is that in any shop where there is a mix of manual and auto-started MQ, the MQ admin must be aware of which approach is used on any given server and move back and forth between different standard procedures fluently. As with any system that depends on human vigilance for quality assurance, some background rate of failure is inevitable. Over time, the chance of queue managers running inside the getty control group and inheriting from its environment approaches certainty. Note that avoiding systemd doesn’t actually make things better since we still end up with all the queue managers running under the getty container rather than a tuned MQ service container. It’s just that doing nothing is the default, and getting from doing nothing to doing something robust and reliable requires an investment in skill and resources so great as to become a showstopper for many MQ shops. Still surprised? Once upon a time, IBM’s strategy for customizing MQ was to expose exit points in the product and trust that each and every customer would have a deep bench of C programmers capable of writing system-level, re-entrant, thread-safe code. Given that legacy it may be understandable why IBM would be surprised to learn that some MQ customers don’t use automation to start and stop MQ. After all, it’s just scripts right? How hard can it be? Man of straw IBM has always advised that the best approach to getting enhancements accepted is to specify the functionality and not the implementation. I’m going to overstep that boundary a bit here in proposing an idealized straw man solution. My complete lack of Linux system programming experience is sure to be revealed by this but we have to start somewhere. Since this is my own blog and not an RFE, I figure I get to do that here. For the sake of this post I’m going to assume that the hypothetical new MQ daemon is called strmqd. It seemed the obvious choice given traditional MQ and UNIX naming conventions. The design would be compliant with systemd expectations: The daemon… Is started by the system at boot. Controls all of the queue managers defined to the server. Manages configuration data such as whether a particular queue manager is intended to run at boot, is enabled or disabled, is singleton or multi-instance, etc. Is capable of figuring out the state of the logical queue manager in order to issue the correct parameters to endmqm. Is capable of progressive, staged queue manager shutdown, including ultimately killing the processes in the prescribed order. Performs something meaningful in response to a refresh instruction from systemd. Provides a command-line interface that MQ administrators can run without requiring root privileges in order to manage configuration data and control queue managers. Provides a command-line interface to set and query the daemon’s configuration parameters individually. Provides a command-line interface to set and query the daemon’s configuration parameters as a group such that the output of the query is capable of being fed back to the set command to restore the daemon configuration. You may have noticed that so far none of these requirements affects existing MQ code. The proposed strmqd is a stand-alone daemon designed on one hand to play well with systemd, and on the other hand to manage queue managers in a way that makes sense to MQ. There are some areas of overlap with existing MQ code that need to be addressed, though. Let’s look at those next. Setting the installation For some reason, the setmqenv command is capable of figuring out which installation a queue manager belongs to but strmqm is not. On a multi-installation node something is therefore obliged to run setmqenv to establish the environment before starting the queue manager. An alternative approach would be for the strmqm command to figure out which installation the requested queue manager belongs to and then, if it isn’t the one from which strmqm was run, set the correct installation and exec the strmqm command from there. Obviously, I don’t know the MQ internals well enough to say whether or why this is a bad idea, but this is exactly the kind of thing IBM objects to when they advise against making recommendations with implementation details. So even though making strmqm dependent on setmqenv may seem like a glaringly obvious deficiency, we have to proceed on the assumption that IBM keeps it this way for reasons we will never know. Fortunately, strmqd is well positioned to take ownership of this task. Until and unless IBM removes the dependency to externally set the installation, the strmqd daemon can figure it out even if that means running setmqenv or crtmqenv and scraping the output. General instrumentation capability Here I am referring the ways in which MQ does or does not interface with external tools to perform administrative or operational tasks or query the state of the queue manager. For example, dmpmqcfg has an option to write each object definition on a single line. This makes it easy to grep through object backup files to identify all objects that meet some criteria. But runmqsc doesn’t have that option so if you want to write a script to parse queue status it will need to be smart enough to correctly parse multi-line, multi-column output. One of these is friendlier to instrumentation than the other. When it comes to instrumentation capability, an honest evaluation of MQ would rate it poorly. Or, as I phrased it in my tweet, “the serviceability gap with MQ start/stop API resembles an open-pit mine.” This post is about interfacing to systemd rather than instrumentation in general, so I will limit my justification for that claim to an abbreviated bullet list. Many long-standing MQ features lack round-trip capability. To give just one example, if the queue buffers are tuned there is no way to capture that in a backup and apply those settings during a restore. Configuration round-trip does not appear to be a primary baseline requirement for new functionality. There are no primitives to manipulate or query namelists. Even IBM has tripped over this. An early version of the FTE installer overwrote the QPUBSUB namelist without first checking for any user-defined values. We wouldn’t need to torture namelists if MQ actually supported user-defined metadata. The runmqsc utility lacks the ability to generate 1-line output, lacks conditional operations, lacks substitutable inserts. Local scripts and agents have no way to manage the multi-instance queue manager’s state at a logical queue manager level. The only way to issue commands to a logical multi-instance queue manager are over a client connection and then only if it is running. Lots of query functions fail if the queue manager is not running, even for singleton queue managers. Many configuration elements and operational states lack query capability. For example, there are commands to determine the MQ data directory if the queue manager is running. If it isn’t running the qm.ini and mqs.ini files must be parsed. An operational example is that there is no way to tell if the queue manager’s in-memory kdb is stale, which is critical when staging certificate renewals. Unfortunately, our hypothetical strmqd daemon needs some of this functionality which is why I mention it here. Ideally, the existing MQ components would be enhanced (some might say fixed) over time to include this functionality. In the meantime, our hypothetical strmqd would need to work around some of these gaps. Native command interception Let’s say IBM grants my wish and delivers strmqd. What should happen when the MQ administrator runs the strmqm or endmqm command? Or any other operational management command? Ideally we don’t want to require the MQ administrator to first figure out if strmqd is in use and then alternate between two completely different sets of operational commands. In an ideal world the native commands would just do the right thing. One approach might be to relocate the native commands and replace them with symlinks to strmqd which knows where to find the original executables if it needs them. I’m sure there are more elegant ways to do this but that’s a good description of the functionality required. This imposes the greatest impact on existing code compared to everything else proposed so far. Yet if it were up to me I would not want to deliver a solution that did not address this. After all, the point of this exercise is to relieve the MQ administrator of the burden of complexity and the cost of a high skill barrier. If the folks in Hursley solve the problem once so that native commands just work regardless of whether strmqd is active or not, the problem is solved for 10,000 customers of all sizes and skill levels Wrapping up One of the benefits of being a consultant is the view of the same product from so many different vantage points. Over time, patterns begin to emerge from commonality of success and failure. Anti-patterns are a peculiar subset that are attractive enough at first glance that many shops tend to gravitate toward them, but that ultimately fail. Given the current capabilities of the product, MQ start/stop instrumentation under systemd qualifies as an anti-pattern. Everyone wants to use it. IBM seems to think we all are. But walk down that path a little way and suddenly you find yourself in a tar pit where the effort to make additional progress is Herculean but for political reasons you can’t easily go back. The result is that many shops either don’t use systemd at all, or else use whatever primitive implementation of an MQ service they had developed at the time they discovered the tar pit and realized they had seriously underestimated the resource commitment to do it right. What is most puzzling is that the IBM dev shop in Hursley hasn’t figured this out yet, especially given their emphasis on using MQ for cloud and virtualization since these use cases practically scream for instrumentation. Instead, IBM implicitly offloads the task of setting up start/stop to the system admin specialists by completely ignoring the subject. When the Linux SA shows up looking for documentation or examples of MQ service files, there are none. Plugging systemd into the search box of the v9.0 Knowledge Center I get back 11 results. Clicking through to the results, none of them actually contain the word systemd so I’m not sure what the hits are supposed to have found. Look for template files or sample scripts in the distribution, there aren’t any. The one thing we did have for a while, SupportPac MSL1, was quietly withdrawn without replacement. When I asked Mark Taylor why at the last MQTC he said “because it didn’t work.” Apparently, IBM found themselves in the same tar pit as everyone else except that they actually were able to back out, erase all the trail marks they had blazed along the way, and pretend the whole thing never happened. My hope is that IBM would implement strmqd or equivalent functionality so that when we create pattern files to provision MQ into the cloud, into containers, or into images spun up on demand, we can bake start/stop functionality into the virtualized host without requiring lots of customization in root-owned files or requiring root escalation. In the meantime, keep your eyes open for tar pits.
Driving MQ admin cost, defects to near-zero
The theme of my sessions at this year’s MQTC (and hopefully also at IBM Think if they are accepted) is cloud and virtualization, if you are reading the abstracts. If you come to the session you find it’s really about … Continue reading →
The theme of my sessions at this year’s MQTC (and hopefully also at IBM Think if they are accepted) is cloud and virtualization, if you are reading the abstracts. If you come to the session you find it’s really about designing architecture around configuration management and tools with the specific intent of driving administrative overhead burden and defects down to near zero. So it was a bit distressing yesterday when during the demo a string of errors cascaded across the screen. Unless you are into schadenfreude, in which case watching my live demo auger into the ground might have been fun for you. But in the end, the event more proves my point rather than invalidating it. Here’s why. I make a point in the demo to tell people that this methodology does not in itself get you to zero defects. The point is to constantly drive down toward zero over time. If you automate a defect you get lots of defects. They are repeatable, consistent, reliable and faithful to the spec, but still defects. Or in my case during the demo, if you fail to anticipate basic error conditions, sooner or later they sneak up to bit you. It’s impossible to anticipate everything so getting an error does not condemn the approach, so long as it is addressed and the fix captured in the tool. The problem I ran into was that I had gutted the Production scripts I used at the last few client engagements, in order to remove the confidential parts and to get the demo to fit into an hour. Amongst the changes, I removed the complex logic that resolved the queue manager’s home directory regardless of where it was mounted and hard-coded /var/mqm/ followed by the QMgr name. That worked great until suddenly, crtmqm started generating queue managers whose home directory was QMGRNAME.0001. Naturally, all the parts of the build script that relied on that began to fail. Live. During the demo. D’oh! This exact type of failure will happen repeatedly over time but it’s normal and part of the process. Sometimes it will be because you overlooked something, as in my case. Other times it will be because the product features or behavior change and the tools need to accommodate that. Other time it is for no other reason than that people find the most creative ways to break software tools. But one way or another, the tool will eventually break or defects will be introduced. Under this methodology the response is to keep iterating fixes into the tool when this happens to constantly push the defect rate lower. In my case, the script now runs dspmqfls to have MQ resolve file locations so it should never fail for that reason again. Naturally, that version of the script is posted to S3 so if you are trying out the tooling then the s3sync.ksh script will probably already have pulled the new bootstrap script version down to your local storage without you even realizing it. If for some reason MQ starts creating versioned QMgr directories you won’t get the failures I had, probably won’t even notice. And THAT is what the methodology and tools are all about. If you are looking for the slides, go to the Links page.
My two sessions from this year’s MQTC are posted: MQ Automation: Config Management Using Baselines, Patterns and Apps Take the grunt work out of MQ configuration management for virtualization, cloud, and large networks by applying a layered approach. This session … Continue reading →
My two sessions from this year’s MQTC are posted: MQ Automation: Config Management Using Baselines, Patterns and Apps Take the grunt work out of MQ configuration management for virtualization, cloud, and large networks by applying a layered approach. This session will introduce the concept of building an MQ configuration from a baseline, then defining a class of service with a pattern layer, and finishing off with application configurations. This modular approach dramatically improves consistency, quality, and flexibility while greatly reducing cost. In compliance environments it provides a definitive as-specified configuration to which the as-running state can be reconciled at intervals or in near-real time. A basic script framework for implementing this system will be reviewed as well. MQ Automation: Config Management using Amazon S3 The central server needed to set up an MQ configuration Management system turns out to be a consistent showstopper, but with a few pennies and a few scripts you can use Amazon Simple Storage. This session introduces scripts that automate QMgr builds with a local shell script that queries a flat-file configuration database stored in the cloud. It’s dirt cheap and super simple yet can reduce the time and cost of building MQ nodes while improving quality and consistency. Note: I created a dedicated user for the conference and am supplying the ID and key in the session materials. Download the slides so you can cut-and-paste the commands to install the AWS metadata files.
In case you hadn’t noticed yet, IBM has quietly changed the format of the stash file so that the various unstash programs no longer work. In this post I’ll discuss some of the security implications of that change and, since … Continue reading →
In case you hadn’t noticed yet, IBM has quietly changed the format of the stash file so that the various unstash programs no longer work. In this post I’ll discuss some of the security implications of that change and, since I never quite grew up, also channel Sean Penn’s Spiccoli from Fast Times at Ridgemont High and make a lot of stash jokes. As Spiccoli might say, “Dude, IBM broke my stash!” Dude, what’s a stash file anyway? First, let’s have a look at the security model of the MQ keystore. The KDB file contains all of the certificates that the queue manager uses. Some of these will be the certificates that it trusts such as CA signer certs and self-signed certs. Some of these will be certificates the queue manager “owns” and presents to the outside world to authenticate it’s identity. These last are called “personal certificates” and are the only certificates in the keystore that contain a private key. Among the files that constitute the keystore, the file with a .kdb extension is the container for all of the certificates. Like the certificates themselves, the keystore is wrapped in a layer of cryptography. The stash file holds the password that unlocks the KDB so that the queue manager or the admin commands can use it. If there’s no stash file you are going to have a really bogus day. Your stash is showing The security model for the kdb and stash files is based on the fact that for either file to be useful it must be possible to reverse the encryption and render the contents of the file in usable form. If it is possible for MQ to do so then it is possible for anyone to reverse engineer the encryption. In fact, the unstash.pl script and its variants that have been floating around the Internet for many years serve exactly that purpose for files using the old stash file format. Someone with the unstash.pl program can find out what the password is on the old stash files. Woah, your stash is showing! Serious bummer, dude. Or is it? The first question to ask is what’s the threat of a working unstash.pl program? In order to use it, an attacker has to first be able to read the stash file and its parent directory. That means the attacker can obtain the stash and entire keystore bundle. Even if they don’t know the password, they don’t need to. They can drop the files into their own queue manager and since it knows how to parse stash files and keystores the attacker can spoof the real queue manager’s identity. Or they can run GSKit commands against it to dump the certificates, including the personal cert with its private key/ In the digital world to see your stash is to be able to steal your stash, dude. That’s like snapping your board in two on a gnarly wave and that could ruin your whole day. So, like, why bother with a stash file, anyway? For operational purposes the string of binary characters that makes up the stash file is the password. An attacker doesn’t need the plaintext password to open the keystore or impersonate the queue manager, they just need the file itself. Encrypting the stash file is not about secrecy. It’s about operational readiness. Unless you want the queue manager to hang and wait for a human to type the password every time it starts,it needs a way to get at the unencrypted password and for most of us that’s the stash file. The purpose of encrypting the stash file is to allow MQ to verify its integrity each time it is opened. That’s it. So what, exactly is the risk if the plaintext password can be known by an attacker? Well, it’s that they know the password. Is that in itself dangerous knowledge? Only if the password is reused in a context where humans have to know it and enter it interactively. You don’t want the keystore password to be the same as the mqm account password, for example. If that’s a real security threat in your shop I have a better way to address it: don’t reuse passwords in general and in particular don’t mix interactive and operational passwords. Any password that must be decryptable can be known. That threat can never be adequately addressed simply by increasing the difficulty of decrypting the password. That threat can be addressed just by making sure that password is not valuable in any other context. Don’t use MQ keystore passwords for login accounts, the app server, the database, etc. Whoa dude, that’s heavy. You saying I’ve been risking my stash all this time, dude? Back in the day, the operator was required to use the -pw option on all of the GSKit commands and type the password into the command line or the program prompt. From a security perspective this was horrible. If you wanted to script the operations then the password had to be on the command line, which made it show up in plaintext in the process table when the command ran. This approach also forced people to observe some sort of password management discipline in order to get the password when they needed to perform maintenance on a keystore. As is entirely predictable in this situation, in most cases this entailed wide reuse of passwords, improper sharing of passwords in emails and sticky notes on monitors, and lots of other practices which were terribly unsecure but which reduced the cost of administrative overhead to something the MQ admin could live with. Even in the best case scenario in which keystore passwords were never reused in other contexts, the need for any password management manifests as bad password management in many cases, and that erodes the overall security discipline of the shop. Woah, no way dude! Yes, way. What am I supposed to do with my stash? <img loading="lazy" decoding="async" class="size-full wp-image-1036 alignright" src="https://t-rob.net/wp-content/uploads/2017/07/Spicolli3.jpg" alt="" width="270" height="330" srcset="https://t-rob.net/wp-content/uploads/2017/07/Spicolli3.jpg 270w, https://t-rob.net/wp-content/uploads/2017/07/Spicolli3-123x150.jpg 123w, https://t-rob.net/wp-content/uploads/2017/07/Spicolli3-245x300.jpg 245w" sizes="auto, (max-width: 270px) 100vw, 270px" />In the new world of GSKit and MQ all of the keystore commands offer the -stashed option so the administrator never needs to know the plaintext password. The commands unwind the stash file using the same method that the queue manager does. Since, as we noted earlier, to read the stash file is to be able to exploit it, there is zero additional risk imposed on the operations side of the security model. But by eliminating the need for really bad password management discipline, the security on the human process side is greatly improved. Here’s what I do: Remove group and world permissions to the keystore directory and files. Change all my scripts and human processes to use the -stashed option for all commands other than creating the keystore. Generate random passwords for all existing keystores and any new ones. Remove all MQ password content from enterprise password vaults, sticky notes, cheat sheets, and so on. Going down the list, the first thing is to make sure that mqm (or the platform equivalent) is the only account with access to the keystore directory and files. This is one case where the files need to be maintained by the MQ service account so I don’t even allow group access to the files if shop policies allow me to restrict them that far. Unfortunately, the MQ account needs write access to the files at run time so you can’t take the approach of write-locking them at run time and granting write permission only during maintenance. The second bullet is really about eliminating the need for humans to know or care what the keystore password is. We can change the human processes easily enough but sometimes it’s tough to track down all the scripts that depend on using the -pw option or on operator entry of passwords. The 3rd task is about ensuring that no queue keystores are accessible using passwords from the password vault or any of the really bad password management that might have been employed. In the case of bad password management it also means an attacker who does obtain the keystore can’t look at the password and gain any insight. For example, if the password is [companyname][current year] there’s a pretty good chance MQ wasn’t the only platform using this derivation. If the password is 64 random characters, the attacker might assume the shop uses really good password discipline and leave befor discovering that all the databases and app servers use [companyname][current year] as their passwords. After all that’s done, remove all evidence of MQ passwords from cheat sheets, sticky notes, password vaults and anywhere else they were stored. Even old, stale information about passwords is useful to an attacker and cleaning that up is just good security hygiene. Gnarly! IBM is cool after all <img loading="lazy" decoding="async" class="alignleft size-full wp-image-1037" src="https://t-rob.net/wp-content/uploads/2017/07/Spicolli2.jpg" alt="" width="275" height="305" srcset="https://t-rob.net/wp-content/uploads/2017/07/Spicolli2.jpg 275w, https://t-rob.net/wp-content/uploads/2017/07/Spicolli2-135x150.jpg 135w, https://t-rob.net/wp-content/uploads/2017/07/Spicolli2-270x300.jpg 270w" sizes="auto, (max-width: 275px) 100vw, 275px" />Right on, dude! If the new functionality causes a problem it isn’t that IBM broke your stash. Depending on knowing the stash file password and building a security model around that was the problem all along. When unstash.pl stopped working all that did was alert you to the problem and in the end that’s a Good Thing. Thanks, IBM! Keystore maintenance typically doesn’t have a lot of scripts and processes so it should be easy enough to go through the instrumentation and human processes to eliminate dependency on knowing the passwords. But if you want some help with that, it’s the kind of thing I love to work on and I have some availability coming up. Surf’s up!
The security maturity progression in MQ starts with access control. First we isolate MQ Admin access, then add granular user and application access. This class of security control is known as intrusion prevention. After mastering that the next phase includes … Continue reading →
The security maturity progression in MQ starts with access control. First we isolate MQ Admin access, then add granular user and application access. This class of security control is known as intrusion prevention. After mastering that the next phase includes stronger accountability and intrusion detection. These typically include enabling event messages and archiving security-relevant events from the event queues and the error logs. In order to detect security-relevant events and hold people strictly accountable within the access roles defined, the product itself must be further along that continuum than the customer is. In particular, the intrusion detection controls must report all events and do so accurately and these tests show that to not be true in all cases. The issues reported in this post affect the ability of an administrator or auditor to accurately detect security-relevant events and enforce accountability. Since error logs roll over, archiving them is subject to a window within which error logs might roll off and be irretrievably lost. Many shops rely on event messages to provide a level of assurance acceptable to auditors and security-conscious users and these errors show that reliance may be misplaced. Parsing event messages If you’ve read just about anything I’ve ever written about debugging MQ Security then you know I’m a huge fan of SupportPac MS0P, in large part because it formats PCF event messages into human readable form. This is a project of Mark Taylor’s and I’ve publicly thanked him in conference sessions and blog posts profusely. MS0P drastically reduces the administrative overhead for managing MQ security. When the event messages in question are Authorization Events, they provide all relevant information needed to diagnose the event. This includes the User ID making the request, the object against which the request was made, and the options used to make the request. Or at least I used to believe they do so reliably, hence my frequent posts advising users to enable Authorization events and install MS0P. Unfortunately, that’s no longer true in all cases. I do not have the full scope of the problem but I’ve found enough issues to warrant concern, and which I’ve documented here. Testing environment These issues were discovered while testing CONNAUTH and CHLAUTH as part of the security research I’ve written about recently in this blog. That testing relies heavily on MQ Explorer so these reports are in that context, and tested on MQ v9.0.0.0 running on Red Hat Enterprise Linux 7.2 x86-64. I have not validated them on other platforms so your mileage may vary. QMgr +dsp wrong permission reported When connecting via Explorer using an ID that has only +connect +inq on the QMgr, a 2035 is thrown. The event message states that Inquire Queue Manager is not authorized: Event Type : Queue Mgr Event [44] Reason : Not Authorized [2035] Event created : 2016/11/01 19:22:53.01 GMT Queue Mgr Name : ASH Reason Qualifier : Cmd Not Authorized Command : Inquire Queue Mgr User Identifier : addrmap2 In this case, the addrmap2 ID actually does have +inq on the queue manager but does not have +dsp which is the actual cause of the authorization error. This is correctly reported in the error log: 11/01/2016 03:22:53 PM - Process(29252.8) User(mqm) Program(amqzlaa0)Host(localhost.localdomain) Installation(Installation1) VRMF(9.0.0.0) QMgr(ASH) AMQ8077: Entity 'addrmap2' has insufficient authority to access object 'ASH'. EXPLANATION: The specified entity is not authorized to access the required object. The following requested permissions are unauthorized: dsp ACTION: Ensure that the correct level of authority has been set for this entity against the required object, or ensure that the entity is a member of a privileged group. The problem here stems from the naming variants used in the different commands. If you have a connection to a Queue Manager and issue Inquire on that connection handle using the API that requires +inq. If you have access to submit PCF commands, the command that requires +dsp is in PCF-speak known as “Inquire Queue Manager” even though if you types it into MQSC it would be DIS QMGR. So the command reported in the event message is correct but you must know that DIS QMGR in MQSC translates to PCF_INQUIRE_QMGR to know why it is correct. At the same time the permission deficiency reported in the log is correct, even though it doesn’t obviously match the event. Neither of these sources contains the complete set of information. Queue +dsp wrong everything reported When connecting via Explorer using an ID that has only +dsp on a subset of queues, many 2035 errors are thrown – two for each queue. Unfortunately, they do not contain the queue name: Event Type : Queue Mgr Event [44] Reason : Not Authorized [2035] Event created : 2016/11/01 18:55:26.84 GMT Queue Mgr Name : ASH Reason Qualifier : Cmd Not Authorized Command : Inquire Queue Status User Identifier : mqm Obviously, mqm is authorized to inquire queue status, or anything else for that matter. This is a stock Linux queue manager with OAM enabled and running as mqm. Since this one tells us it’s a queue inquiry problem we expect it to show us the queue name but that is missing. MQ Explorer issues a DIS Q(*) and a DIS QSTATUS(*) for each refresh of the queues screen, and this causes MQ to spit out one of these for each unauthorized queue for each of the two commands. In my test queue manager with only a couple of user-defined queues that’s more than 100 235 errors for each repaint of the Explorer queues screen. The AMQERR01.LOG file shows the object name, the correct entity, and identifies +dsp as the missing permission but fails to identify the command that was executed: 11/01/2016 02:55:26 PM - Process(29252.8) User(mqm) Program(amqzlaa0) Host(localhost.localdomain) Installation(Installation1) VRMF(9.0.0.0) QMgr(ASH) AMQ8077: Entity 'addrmap2' has insufficient authority to access object 'SYSTEM.SELECTION.VALIDATION.QUEUE'. EXPLANATION: The specified entity is not authorized to access the required object. The following requested permissions are unauthorized: dsp ACTION: Ensure that the correct level of authority has been set for this entity against the required object, or ensure that the entity is a member of a privileged group. Resource enumeration weakness I used Wireshark to sniff the client traffic in order to update this post. The network capture revealed something else of great interest. Error messages returned to the client are intentionally sparse in order to limit information leaked to an attacker. All the legitimate user is supposed to know is that an authorization error occurred. They don’t get to know things like, for example, the object name if their request had a wildcard. IBM has duly suppressed display of the queue name in the error message returned to the client. Unfortunately, it was also suppressed in the diagnostics emitted by the queue manager. The interesting bit is that network sniff revealed that an attacker gets back a separate 2035 PCF response for every single queue they are unable to display. The queue name is suppressed in the client error message, but the number of them is returned for the asking. we don’t see this because MQ Explorer silently discards the info, but anyone who can run MQ Explorer could as easily run their own Java code and see the same information. How might this be abused? Assume for example that your B2B queue manager uses structured names with customer or branch numbers embedded and I know of one called SALES.ORDER.REQ.12345.QA. If I have enough authorization to issue DIS Q(*) then I can also DIS Q(SALES.ORDER.REQ.*) and discover how many other clients access this node, and how often that population increases or decreases, with a fair degree of confidence. Then when I see it’s increased by one I can call the Help Desk, claim to be the new customer and “gosh this is so hard, what’s the name of that new queue again?” But we both know you don’t really want to wade through millions of spurious 2035 errors. Chances are you either ignore them entirely or else suppress them altogether. That means I’m safe to just enumerate all the queues: DIS Q(SALES.ORDER.REQ.12345.*) DIS Q(SALES.ORDER.REQ.12346.*) DIS Q(SALES.ORDER.REQ.12347.*) DIS Q(SALES.ORDER.REQ.12348.*) ... The first one gives me some insight as to whether your naming convention includes a QL or QR for every QA. Then for each of the other queues I compare the number of hits to the number returned in the first one to learn if that naming convention is consistent and how many other customer numbers there are queues for. Or I can explore the topology of the entire sales system: DIS Q(SALES.*) DIS Q(SALES.ORDER.*) DIS Q(SALES.ORDER.REQ.*) DIS Q(SALES.ORDER.REQ.12345.*) Again, comparing the number of errors in the known queue to the ones returned by the other inquiries gives me an idea of the size and complexity of the system. Or I can dissect the system by known prefix: DIS Q(*) DIS Q(SYSTEM.*) DIS Q(SYSTEM.FTE.*) DIS Q(SYSTEM.BROKER.*) DIS Q(SALES.*) The first command gives me total number of queues. The next three give me a breakdown of how many are SYSTEM.* and whether Broker or FTE queues are defined. The last gives me the one topology I already know about – SALES – which I combine with the number of SYSTEM.* queues and subtract from the total to glean the rough number of remaining user-defined queues. You don’t give up your queue names but knowing the precise numbers of each kind of queue at this level of detail helps convince the Help Desk person that I’m legit. Performance issues There’s also the minor issue that sending back all those statuses can impact performance. One of my customers had thousands of queues in a shared hub environment and strict application isolation. When they grant limited access, their developers see only a handful of queues in Explorer. I’d been assuming they don’t get back errors for all the queues they don’t see. Now I kow that under the covers they get the PCF responses for those authorized queues plus 2x as many 2035 responses as there are non-authorized queues, and that this happens every time Explorer refreshes the queues screen. Multiply that times all the developers, testers and other staff for 80+ active application development projects and the logs are truly useless and roll over quickly. The impact to Tivoli or other monitoring tool that captures those event messages is a firehose of spurious messages which clog up the archive database. That’s why I as the attacker am so confident that you are either actively suppressing those errors or, more likely, are ignoring them entirely. Why do I think you aren’t suppressing them? Easy. There’s no option for “please suppress the spurious security relevant events but show us the legitimate ones” because the attackers overwhelmingly ignore RFC 3514 which requires them to set the Evil Bit specifically for this purpose. (Except for Ben, of course.) Recommendations Short version: Take those event messages with a grain of salt as they don’t show the whole story and at least in the cases I’ve identified they show the wrong user ID. If you really need to enforce authorization, it might be worth while to look at archiving error logs. I may write some scripts for that here soon due to this problem. Also, go vote for RFE 86864: Add object name to Not Auth (Type 4) Command Not Authorized. (Thanks, Morag!) That this is a defect and not an enhancement seems obvious but sometimes the RFE is the faster path. Longer version: I have begun to suspect that IBM lacks a definitive specification for how security in MQ works with regard to AUTHREC, CONNAUTH and CHLAUTH. My testing results matrix show that the authorization decisions have alternated between the process ID and the one that was authenticated by a password, depending on the version and Fix Pack. Similarly, the results reported here show that at the conclusion of CONNAUTH and CHLAUTH there remain two distinct IDs maintained by MQ and different pieces of code with similar function refer to these in different ways. Here’s a few requirements that MQ doesn’t meet but which one would think would be blindingly obvious: The ID that was authenticated MUST be the one that authorizations are evaluated against unless overridden by the MQ Administrator. It should NEVER be the case that the client can override the ID used for authorizations or that the client has more leeway to do so than the MQ administrator, but that’s what we have in all versions that support password-based validation. At the conclusion of CONNAUTH and CHLAUTH there should be one and only one ID used for authorization checking. We saw in the last couple of posts that this is not the case and we see in this post that different parts of the MQ code refer to different ID fields from the connection for reporting purposes. Taken together, these weaken accountability and intrusion detection making it impossible for MQ users to confidently move up the security maturity continuum. Ideally, the original ID may be retained but only for reporting of context. In the example provided here the process ID was mqm but the MCAUSER was addrmap2. If reporting only one ID it must be the one used for authorization checking, in this case addrmap2. If a second ID is reported it should provide context. In this example mqm could be reported as the Real ID and addrmap2 as the Effective ID (borrowing from the same terms as used by flavors of the UNIX OS). Instead, mqm was reported as the ID that failed the auths check. Don’t report spurious security events. If the system supports a function like DIS Q(*) or DIS QS(*) then return only the objects to which the caller authorized and silently skip the rest. If the caller is not authorized to any objects of that type, then return a single error in the log and cut an event message. Make sure the events and error logs reported are accurate and reconcile to the test cases that generate them. There should be no case in which the events and error logs differ in count or content, nor in which an individual event is reported more or less than once. In this case the events tell part of the story and the error logs tell another part and you need both to get all the information. When I lobbied my IBM management to work on the MQ Security team it was specifically to address issues like this in the role of an architect or QA or Dev team lead. Unfortunately, due to geographic and other restrictions the closest I was able to get to that position was product manager and that role had no direct influence over specifications at this level. If I had that opportunity today the first thing I’d do is lay out a clear and unambiguous specification for interaction of CONNAUTH and CHLAUTH that mapped every single element in the test matrix I’ve posted and then expand that to include real/effective IDs and exit interaction. With the losses from the MQ team over the years of Ian Vanstone, Dale Lane, Morag, Dermot, me, and others who focused deeply on security, that area seems to be suffering. Lets hope IBM gets some more security specialist talent deeply embedded on that team. I’d do it as a consultant to tide them over, or as a permanent hire if I actually got to focus on MQ security this time but however it happens this situation needs to get better soon. Look for more posts shortly. I have a backlog generated from the recent round of research and testing that I need to clear. Post updated 20161101 23:17 EDT to include results of network packet capture sniffing that revealed the details of the DIS Q(*) and DIS QSTATUS(*) as well as the resource enumeration issues. Post updated 20161109 13:10 EDT to include reference to Morag’s RFE and fix some typos and unclear wording.