Store and Forward
type: channel
description: A blog about securing and using IBM MQ
date: 2020-04-05T21:08:14Z
update period: hourly
update frequency: 1
update base: 2000-01-01T12:00+00:00
generator agent: https://wordpress.org/
items: MQ Safety Tips for the Pandemic
items: Command Server Gotcha
items: Client-based runmqsc gotcha
items: When deprecated != deprecated
items: Parsing MQ error logs in Splunk
items: MQ start/stop woes
items: Driving MQ admin cost, defects to near-zero
items: MQTC v2.0.17 Sessions
items: Dude, IBM broke my stash!
items: Inaccurate MQ auths event messages

MQ Safety Tips for the Pandemic
type: item
creator: T.Rob
date: 2020-04-05T21:03:28Z
subject: General
subject: Humor
description: A comprehensive guide to surving the pandemic from an #IBMMQ perspective. Continue reading →
encoded: Every day I receive one or more new email memes in my inbox purporting to have the most up to date information available to keep me safe from the virus. Of course, none of these point to authoritative sources and most have at least one egregious error in them, begging the question of why anyone believes them in the first place. Worse, none of these informal memes, at least among those I’ve personally seen, deal with #IBMMQ specifically, which I think is a damned shame and one I hope to correct here and now. I have rounded up the most relevant health and safety tips for MQ available from the CDC (Centers for Data Control), Irene Ken whose daughter is an assistant Prof at Johns Hopkins, a friend who is a distant relative of Dr. Anthony Fauci, and IBM’s Viral Marketing Department. Here then I present to you my comprehensive list of IBM MQ Tips for the Pandemic. Detection Virus scan your messages! Unlike the real world in which testing requires atoms, our scanners are made out of software and indefinitely replicable. So when I say “we have enough scanners for everyone” it is provably true. Border patrol: If your internal QMgr attempts to break quarantine and contact another country, it probably has a virus. Use Advanced Message Security to hash message data. Altered messages will automatically be quarantined in the Protection Queue for their own protection. And yours. Check the temperature of your servers regularly. Viruses often cause servers to run hot. Your QMgr may have a virus if the Dead Queue fills rapidly or overflows. Prevention Use SOAP. Except for PCF, of course. But for everything else, use SOAP. When using SOAP, process messages for at least 20 ms. REST often to boost the immune system. In fact, if you can use SOAP over REST that’s even better. Mask sensitive data! And not just in the public domain. The CDC (Centers for Data Control) which has previously been ambiguous on the matter now recommends masking sensitive data at all times. (Or was that the PCI Co…

Command Server Gotcha
type: item
creator: T.Rob
date: 2018-05-14T14:45:20Z
subject: IBMMQ
subject: IIB
subject: MQ AMS
subject: MQMFT
subject: News
description: The title of my last post, Client-based runmqsc gotcha, was a bit of a misnomer. Although I stumbled onto it using runmqsc client, the root cause of the issue really lies with the Command Server. That means anything that drives … Continue reading →
encoded: The title of my last post, Client-based runmqsc gotcha, was a bit of a misnomer. Although I stumbled onto it using runmqsc client, the root cause of the issue really lies with the Command Server. That means anything that drives commands through the Command Server can run into this issue, making it a much bigger deal than I originally thought. IBM has replied to the PMR so I’ll discuss that along with a suggested strategy to mitigate the risk and hopefully avoid Production outages. How it breaks Root cause If a PCF command causes the Command Server to return more result messages than the MAXDEPTH of the reply-to queue, the Command Server may return partial result sets and silently abort further processing. Usually the client consuming the responses can get them fast enough that a result set only slightly larger than MAXDEPTH will succeed. However, as the result set size increases beyond that the failure rate increases. In my case when the Command Server was asked to return a result set of 4005 objects into a reply-to queue with MAXDEPTH(3000) the failure rate was about 60%. Differing behaviors I saw three different failure behaviors in runmqsc. In the best case runmqsc printed an “AMQ8101: WebSphere MQ error (805) has occurred” to indicate a problem. But it also sometimes printed “MQSC found the following response to a previous command on the reply q” in the middle of the result set. Based on the numbers it appears in these cases to have lost some varying number of the non-persistent responses between when the queue filled and enough free space opened up for the Command Server to continue. Since the reply queue is dynamic it should be impossible to find a “response to a previous command” so this suggests that the Command Server has at least some retry logic though it doesn’t appear to be very robust.1 The other behavior I saw was that no indication of a problem occurred other than receipt of an incomplete result set. This suggests that some of the non-persistent resp…

Client-based runmqsc gotcha
type: item
creator: T.Rob
date: 2018-05-03T17:57:26Z
subject: IBMMQ
subject: MQTC
description: Most shops running more than a handful of queue managers end up writing a script or two that sends commands to runmqsc and parses the responses. Until recently the options were to run those scripts locally or to access remote … Continue reading →
encoded: Most shops running more than a handful of queue managers end up writing a script or two that sends commands to runmqsc and parses the responses. Until recently the options were to run those scripts locally or to access remote queue managers using the MQ network. Neither of these options is ideal. Use of local scripts requires maintaining a code base distributed across a variety of hosts and often disparate platforms. Sending runmqsc commands across the MQ network solves those problems but requires a security model in which at least some of the adjacent queue managers can administer each other. The new MQ Client capabilities in runmqsc give us the best of both worlds. We can maintain a single set of scripts in a central location and use them to connect directly to each of the queue managers using MQ client. Some functions that require correlation of state across multiple queue managers are actually easier to build using MQ client. There is, however, a gotcha that anyone using client-based runmqsc should be aware of. I’ll explain that here along with some techniques to mitigate the issue and feedback from the PMR I opened. Workflow The steps to complete a client runmqsc call are as follows: The runmqsc component is executed in client mode on the remote node and requests a connection to the target queue manager. The queue manager’s Message Channel Agent process amqrmppa receives the connection… …creates a dynamic reply-to queue based on SYSTEM.MQSC.REPLY.QUEUE… …and places the command on SYSTEM.ADMIN.COMMAND.QUEUE. The Command Server process amqpcsea receives the command from the queue. The Command Server places the responses on the dynamic Reply-To queue. The Channel Agent transmits the messages over the network. The runmqsc client receives and displays the response set. runmqsc–>amqrmppa–>amqpcsea–>amqrmppa–>runmqsc Note that runmqsc connecting to a local queue manager might also use the command server. Although it could directly access MQ objects in this scenario, i…

When deprecated != deprecated
type: item
creator: T.Rob
date: 2018-01-19T19:26:44Z
subject: IBMMQ
subject: Security
subject: WMQ Security
subject: Admin
subject: Architecture
subject: Best Practices
subject: ciphersuite
subject: deprecated
subject: IBM MQ
subject: NULL
subject: Recommended Practices
subject: SSL
subject: TLS
subject: WebSphere MQ Security
subject: WMQ
description: Every now and then we hear that a cipher algorithm has fallen to a new cracking technique. This cascades into a new round of deprecating any ciphersuites that rely on the newly cracked algorithms. Over the years we’ve moved from … Continue reading →
encoded: <img decoding="async" class="alignnone size-full wp-image-1096" src="https://t-rob.net/wp-content/uploads/2018/01/2018-01-19_14-15-13.jpg" alt="" width="600" height="150" srcset="https://t-rob.net/wp-content/uploads/2018/01/2018-01-19_14-15-13.jpg 600w, https://t-rob.net/wp-content/uploads/2018/01/2018-01-19_14-15-13-150x38.jpg 150w, https://t-rob.net/wp-content/uploads/2018/01/2018-01-19_14-15-13-300x75.jpg 300w" sizes="(max-width: 600px) 100vw, 600px" /> Every now and then we hear that a cipher algorithm has fallen to a new cracking technique. This cascades into a new round of deprecating any ciphersuites that rely on the newly cracked algorithms. Over the years we’ve moved from SSL to TLS, from DES to 3DES, from MD5 to SHA, and so on. The list of items on MQ’s Deprecated Ciphers page grows slowly but relentlessly over time. But not all of the ciphersuites that IBM has deprecated are broken. So why are they on the list and should we use them? This post will attempt to shed some light on those questions. Non-deprecated deprecation Sprinkled among all the other deprecated ciphers are a few with no deprecated component As of this writing, these include: ECDHE_ECDSA_NULL_SHA256 ECDHE_RSA_NULL_SHA256 TLS_RSA_WITH_NULL_NULL TLS_RSA_WITH_NULL_SHA256 Note that all of the component protocols—ECDHE, ECDSA, TLS, RSA, and SHA256—are present in one or more ciphersuites that are not deprecated. On the other hand, none of the non-deprecated ciphersuites contain a NULL component. The obvious conclusion is that these entries made the list because of the NULL components of their respective ciphersuites. NULL hasn’t been cracked because there’s nothing to crack. So these aren’t deprecated because of a crypto weakness but rather because of a policy that discourages the use of NULL ciphers in a ciphersuite. Is that really a good thing? The case for null ciphers TLS provides three different security functions: authentication, privacy (encryption), and integrity (hashing). The ciphersui…

Parsing MQ error logs in Splunk
type: item
creator: T.Rob
date: 2017-12-19T02:39:56Z
subject: IBMMQ
description: A how-to tutorial and code samples for parsing #IBMMQ error logs in @Splunk. #IBM @IBMChamp Continue reading →
encoded: A growing number of my clients are deploying IBM MQ on Amazon EC2 instances and a common need that I see emerging is for instrumentation and tooling. When the MQ instance is ephemeral, deploying instances on demand and decommissioning just as suddenly, lots of things the MQ Admin used to do by hand need to be automated. This includes build-time things such as defining objects, run-time tasks like enabling or disabling queues in the cluster, and forensic capabilities such as archiving error logs. It is this last item that concerned a recent customer. Their main requirement was to ingest MQ error logs in real time, or at least close to it, so those logs would survive death of the virtual host on which they were generated. Getting Splunk to ingest the logs was ridiculously easy. Just define the log files as a Splunk data input and immediately they become available through the Splunk search interface. That’s all well and good if all you want to do is browse through the logs or search them for particular error codes. To get the benefit of Splunk analytics requires the error logs to be parsed into fields. Then instead of merely searching for error codes you already know about, you can ask Splunk to show you a report of all the error codes sorted by prevalence, by frequency over time, or even which ones are the rare outliers. All the analytic capabilities are usable once the fields are parsed. Better yet, parse logs from many queue managers and now you can spot trends or pick out nodes that are showing early signs of distress. That’s really useful stuff and Splunk provides it right out of the box, but only for log types it knows how to parse. So let’s teach it to parse IBM MQ error logs, shall we? MQ error log structure Before wading into the transform code, let’s look at a couple of typical IBM MQ error log entries. The first of these is reporting an error. The second one reports a normal channel startup. 12/18/2017 04:12:54 PM - Process(99475.1) User(mqm) Program(runmqch…

MQ start/stop woes
type: item
creator: T.Rob
date: 2017-10-22T21:54:27Z
subject: Fail
subject: IBMMQ
description: After I tweeted a link to an IBM blog post on how to start and stop IBM MQ using systemd, an IBMer responded to say “it surprised me to hear that some #IBMMQ customers have to manually restart their QMs … Continue reading →
encoded: After I tweeted a link to an IBM blog post on how to start and stop IBM MQ using systemd, an IBMer responded to say “it surprised me to hear that some #IBMMQ customers have to manually restart their QMs when the box comes up.” My reply was brutally frank: “Should be no surprise – the serviceability gap with MQ start/stop API resembles an open-pit mine. As a result most shops either don’t do it well, or don’t do it at all. Mortgage payments on the technical debt needed here desperately.” Not wanting to leave that hanging out there with no explanation, this post describes in excruciating detail what’s wrong. Hopefully, that’s the first step to getting it fixed. So you want to start MQ? Sounds easy enough, right? Just write a systemd service to kick off strmqm and move on, yes? Not so fast! Unless there’s a default queue manager, the service has to at least know the name of the queue manager to start and if there is more than one queue manager, the service needs to know about all of them. The IBM blog post addresses this by using systemd templates. There’s one generic file with the ability to pass in a substitution parameter. Each queue manager gets a symbolic link to the template file. The link file name contains the queue manager name which is the source of the substitution. So, assuming we have a good template and several links we are good to go, yes? Not so fast! Remember that, like any other boot-level service, systemd runs as root and its configuration files must therefore not be writable by anyone who isn’t root. That means if the recommend practice of giving every queue manager a unique name is followed, the MQ administrator must engage the UNIX/Linux System Administration team for every invocation of crtmqm or dltmqm. So you want to start multi-installation MQ? Multiple MQ installations on the server breaks the assumption that starting a queue manager is as simple as running strmqm. Now the systemd service is obliged to properly set the environment for each of…

Driving MQ admin cost, defects to near-zero
type: item
creator: T.Rob
date: 2017-09-27T16:54:02Z
subject: Events
subject: IBMMQ
subject: MQTC
description: The theme of my sessions at this year’s MQTC (and hopefully also at IBM Think if they are accepted) is cloud and virtualization, if you are reading the abstracts. If you come to the session you find it’s really about … Continue reading →
encoded: The theme of my sessions at this year’s MQTC (and hopefully also at IBM Think if they are accepted) is cloud and virtualization, if you are reading the abstracts. If you come to the session you find it’s really about designing architecture around configuration management and tools with the specific intent of driving administrative overhead burden and defects down to near zero. So it was a bit distressing yesterday when during the demo a string of errors cascaded across the screen. Unless you are into schadenfreude, in which case watching my live demo auger into the ground might have been fun for you. But in the end, the event more proves my point rather than invalidating it. Here’s why. I make a point in the demo to tell people that this methodology does not in itself get you to zero defects. The point is to constantly drive down toward zero over time. If you automate a defect you get lots of defects. They are repeatable, consistent, reliable and faithful to the spec, but still defects. Or in my case during the demo, if you fail to anticipate basic error conditions, sooner or later they sneak up to bit you. It’s impossible to anticipate everything so getting an error does not condemn the approach, so long as it is addressed and the fix captured in the tool. The problem I ran into was that I had gutted the Production scripts I used at the last few client engagements, in order to remove the confidential parts and to get the demo to fit into an hour. Amongst the changes, I removed the complex logic that resolved the queue manager’s home directory regardless of where it was mounted and hard-coded /var/mqm/ followed by the QMgr name. That worked great until suddenly, crtmqm started generating queue managers whose home directory was QMGRNAME.0001. Naturally, all the parts of the build script that relied on that began to fail. Live. During the demo. D’oh! This exact type of failure will happen repeatedly over time but it’s normal and part of the process. Sometimes it will …

MQTC v2.0.17 Sessions
type: item
creator: T.Rob
date: 2017-09-25T17:32:23Z
subject: General
description: My two sessions from this year’s MQTC are posted: MQ Automation: Config Management Using Baselines, Patterns and Apps Take the grunt work out of MQ configuration management for virtualization, cloud, and large networks by applying a layered approach. This session … Continue reading →
encoded: My two sessions from this year’s MQTC are posted: MQ Automation: Config Management Using Baselines, Patterns and Apps Take the grunt work out of MQ configuration management for virtualization, cloud, and large networks by applying a layered approach. This session will introduce the concept of building an MQ configuration from a baseline, then defining a class of service with a pattern layer, and finishing off with application configurations. This modular approach dramatically improves consistency, quality, and flexibility while greatly reducing cost. In compliance environments it provides a definitive as-specified configuration to which the as-running state can be reconciled at intervals or in near-real time. A basic script framework for implementing this system will be reviewed as well. MQ Automation: Config Management using Amazon S3 The central server needed to set up an MQ configuration Management system turns out to be a consistent showstopper, but with a few pennies and a few scripts you can use Amazon Simple Storage. This session introduces scripts that automate QMgr builds with a local shell script that queries a flat-file configuration database stored in the cloud. It’s dirt cheap and super simple yet can reduce the time and cost of building MQ nodes while improving quality and consistency. Note: I created a dedicated user for the conference and am supplying the ID and key in the session materials. Download the slides so you can cut-and-paste the commands to install the AWS metadata files.

Dude, IBM broke my stash!
type: item
creator: T.Rob
date: 2017-07-28T16:27:01Z
subject: IBMMQ
subject: Security
description: In case you hadn’t noticed yet, IBM has quietly changed the format of the stash file so that the various unstash programs no longer work. In this post I’ll discuss some of the security implications of that change and, since … Continue reading →
encoded: In case you hadn’t noticed yet, IBM has quietly changed the format of the stash file so that the various unstash programs no longer work. In this post I’ll discuss some of the security implications of that change and, since I never quite grew up, also channel Sean Penn’s Spiccoli from Fast Times at Ridgemont High and make a lot of stash jokes. As Spiccoli might say, “Dude, IBM broke my stash!” Dude, what’s a stash file anyway? First, let’s have a look at the security model of the MQ keystore. The KDB file contains all of the certificates that the queue manager uses. Some of these will be the certificates that it trusts such as CA signer certs and self-signed certs. Some of these will be certificates the queue manager “owns” and presents to the outside world to authenticate it’s identity. These last are called “personal certificates” and are the only certificates in the keystore that contain a private key. Among the files that constitute the keystore, the file with a .kdb extension is the container for all of the certificates. Like the certificates themselves, the keystore is wrapped in a layer of cryptography. The stash file holds the password that unlocks the KDB so that the queue manager or the admin commands can use it. If there’s no stash file you are going to have a really bogus day. Your stash is showing The security model for the kdb and stash files is based on the fact that for either file to be useful it must be possible to reverse the encryption and render the contents of the file in usable form. If it is possible for MQ to do so then it is possible for anyone to reverse engineer the encryption. In fact, the unstash.pl script and its variants that have been floating around the Internet for many years serve exactly that purpose for files using the old stash file format. Someone with the unstash.pl program can find out what the password is on the old stash files. Woah, your stash is showing! Serious bummer, dude. Or is it? The first question to ask is what’s…

Inaccurate MQ auths event messages
type: item
creator: T.Rob
date: 2016-11-01T20:21:35Z
subject: IBMMQ
subject: News
subject: Security
subject: WMQ Security
subject: authorization
subject: AUTHREC
subject: bugs
subject: Events
subject: logs
subject: security
subject: setmqaut
subject: WebSphere MQ Security
description: The security maturity progression in MQ starts with access control. First we isolate MQ Admin access, then add granular user and application access. This class of security control is known as intrusion prevention. After mastering that the next phase includes … Continue reading →
encoded: The security maturity progression in MQ starts with access control. First we isolate MQ Admin access, then add granular user and application access. This class of security control is known as intrusion prevention. After mastering that the next phase includes stronger accountability and intrusion detection. These typically include enabling event messages and archiving security-relevant events from the event queues and the error logs. In order to detect security-relevant events and hold people strictly accountable within the access roles defined, the product itself must be further along that continuum than the customer is. In particular, the intrusion detection controls must report all events and do so accurately and these tests show that to not be true in all cases. The issues reported in this post affect the ability of an administrator or auditor to accurately detect security-relevant events and enforce accountability. Since error logs roll over, archiving them is subject to a window within which error logs might roll off and be irretrievably lost. Many shops rely on event messages to provide a level of assurance acceptable to auditors and security-conscious users and these errors show that reliance may be misplaced. Parsing event messages If you’ve read just about anything I’ve ever written about debugging MQ Security then you know I’m a huge fan of SupportPac MS0P, in large part because it formats PCF event messages into human readable form. This is a project of Mark Taylor’s and I’ve publicly thanked him in conference sessions and blog posts profusely. MS0P drastically reduces the administrative overhead for managing MQ security. When the event messages in question are Authorization Events, they provide all relevant information needed to diagnose the event. This includes the User ID making the request, the object against which the request was made, and the options used to make the request. Or at least I used to believe they do so reliably, hence my frequent posts…