My IMPACT sessions are in process of being reassigned so I’ve adjusted the schedule on the previous blog post.  I left “Meet The Experts” on the schedule, I’ll just be in the audience this time around.  I’ll also be there for the book signing and there are 100 copies printed (including the updated typos) so come by the table Tuesday at Noon.  I believe that Neil Casey will also be in attendance and at the book signing.

The new business is IoPT Consulting.  If you’ve heard of Internet of Things, the name of the business refers to Internet of People & Things.  My biggest gripe with the first wave of IoT devices is that they neglected to consider the people who would use them.  The cool factor of a light switch, door lock, web cam or other device controlled from your phone wears off pretty quick when you have to have 50 apps to control 50 categories of device and they don’t talk to one another.  Of course IoT has no appeal at all when your Internet connection goes dark and the house blue-screens.  I’d like to build an Internet of Things that values the people who own those things and, by the way, WebSphere messaging and MQTT are one of the best ways to do that.

I’ll be available as of May 13 as an independent or, for customers with preferred vendor rosters, through one of several established firms.  I can provide short- or long-term engagements for architecture design, performance tuning, outage resolution, migration, staff augmentation, and of course security.  Lots of security.  Call me for details st 704-443-TROB or see me at IMPACT.

Well, now I am.

IBM will send me pretty much get to anywhere in North America so long as I can meet with two or more customers (but not at the same time).  If I line up 4 or more customers I can visit other continents.  So if you want to meet in person to talk security…or clustering, architecture, high availability, migration, AMS, FTE, MQTT, identity, privacy, Internet of Things, whatever, let me know.  Maybe I can find a few others in your area and we can work something out.

Upcoming trips include:

• Monday evening, April 8: PDCNYC
• April 9 – 10: Customer visits, Jersey City/NYC area.
• April 28 – May 3: IMPACT, Las Vegas (Book signing, Tuesday at Noon.)
• May 6 – 9: Internet Identity Workshop #16, Mountain View, CA.

TSM-2018: Meet the Experts: IBM Messaging
Session Type:  Meet the Experts
Date/Time:  Mon, 29/Apr, 04:00 PM – 05:00 PM
Room:  Venetian – San Polo 3401 (Zone D)
——————————————

Secure Messaging Scenarios with WebSphere MQ
Session Type: Book Signing
Date/Time:  Tue, 1/May, 12:00 PM – 1:00 PM
Room: Conference Book Store
——————————————

BPB-3218: Better Access Control and Security Using a Single Portal
Session Type:  Birds of a Feather
Date/Time:  Wed, 1/May, 12:00 PM – 12:45 PM
Room:  Venetian – Lando 4301B
Co-presenter(s):  Peter D’Agosta, Avada Software
——————————————

TSM-2018: Meet the Experts: IBM Messaging
Session Type:  Meet the Experts
Date/Time:  Thu, 2/May, 08:45 AM – 09:45 AM
Room:  Venetian – San Polo 3401 (Zone D)
——————————————

I should also mention that these are my observations based on interactions with many customers and not necessarily the official position of IBM.

I refer specifically to PCI-DSS requirement 8.4 “Render all passwords unreadable during transmission and storage on all system components using strong cryptography.”  The problem with this is that it doesn’t distinguish between the two types of password, each of which have very different security requirements.  However because under the PCI-DSS a password is a password, QSA’s and implementers often treat them all equally.  In the few cases where the different types of password are treated according to their unique requirements, the customer has no assurance of passing subsequent assessments unscathed since the next assessment may interpret the standard differently.

The primary use case for passwords, and the one that the PCI-DSS appears to address, is the one I outlined in the first paragraph: that of a networked service to which users log on.  Best practices are to encrypt the account password with many iterations of a one-way salted hash, storing the resulting value.  Subsequent login attempts transform the password presented in the login request using the same algorithm and then compare the result to the stored value.  This is the classic case of encrypting stored passwords.  Note that there is no requirement in this use case for the stored password to ever be rendered in plaintext.  I will refer to this use case as an inbound authentication request because the thing storing the passwords is receiving an access request.

The second use case for passwords is where an application makes an outbound access request of something else.  Typical examples are when an application authenticates to a remote LDAP server or requests a connection to a JMS message broker.  In this use case, the application presenting the ID and password must render them in plaintext in order to make the access request.    I will refer to this use case as an outbound authentication request because the thing storing the passwords is requesting access to something else.

Note that in the inbound access request use case the stored password must never be rendered in plaintext, while in the outbound access request use case the stored password must always be rendered in plaintext.  If your intuition tells you that these two use cases differ greatly in their security requirements, and that attempting to treat them the same would result in poor security, then your intuition would be correct.  However the wording of the PCI-DSS fails to differentiate between the two use cases and customers frequently report that their QSA has specified remediations that are complex, expensive, and either fail to improve security or actually make it worse.  The justification in these cases is that “the PCI-DSS requires it.”

Because the PCI-DSS does not distinguish between these two use cases for passwords, the mitigations specified fail to address either case adequately and lumps them all together as a requirement for “encryption.”  In addition to treating the outbound side as a seperate use case, the standard should refine the use case on the inbound side to specify non-reversible encryption, perhaps borrowing from OWASP.

I propose that the standard distinguish between passwords for inbound access requests versus those for outbound access requests.  The current best practices of salted, hashed passwords using strong, one-way encryption would be associated with inbound requests.  There hasn’t been a discussion on how best to secure passwords for outbound requests since we have not, as yet, seen fit to talk about them separately.  That discussion should take place as soon as possible in order to arrive at a consensus and codify that into the standard.

In order to facilitate that discussion,  consider a JEE server that is started when its host reboots.  The JEE server then requests a connection to a database using an ID and password.  Although the database stores a hashed version of the password, the JEE server must render it in plaintext in order to present it to the database.  Usually the JEE server retrieves its ID and password from the filesystem, either in an ini file, in a keystore, in boot scripts or in some other location.  Sometimes that file, or perhaps just the password in the file, is encrypted.  In that case, how does the JEE server render it into plaintext?  A secondary password is required and it, too, must be stored somewhere.  In order to comply with the standard, many QSA’s tell their customer that this secondary password must also be encrypted.  However because that encryption must be reversed, doing so requires a tertiary password and that must also be stored somewhere.

As should be clear by now, there are no number of iterations that eliminate the need for a plaintext password somewhere in the file system on the outbound side of the authentication request.  However, many assessmors arbitrarily force customers to make one or more iterations or to pass this requirement up to their software vendors.  This always results in a password in the clear somewhere and trivial reversal of the original, but so long as the original is not in plaintext, the assessment is satisfied because “the standard says to do it that way.”

As an alternative, some users and vendors opt for obfuscation of the password by running it through an encryption process using a key compiled into the application.  This meets the letter of the standard but is trivially reversed.  Others derive a key at runtime, but again this is trivially reversed.  If you know how the key is derived and have access to the runtime environment of the server, you can decrypt the password.  Some customers encrypt the filesystem on which the password file exists.  This is great if someone steals the disk but the standard seeks to address a completely different threat – that of data exfiltration from a live system.

Subjecting passwords to multiple layers of reversible encryption where the ultimate password is stored in the clear on the file system increases complexity and cost and provides negligible additional protection.

Furthermore, after having obfuscated the passwords, users frequently believe them to be secure and fail to protect them properly in the filesystem or in backups. The end result in many cases is a system that is less secure than when the outbound authentication passwords were in plaintext because in that case the user was cognizant of the need to protect them in the filesystem.

My proposal is as follows:

• Distinguish in the PCI-DSS between one-way encrypted passwords on the inbound side of the access request vs. reversibly encrypted passwords on the outbound side of the access request.
• Incorporate the existing best practices for password storage on the inbound side of the access request.  Specifically, using salt and multiple hash iterations.
• Perform a formal threat analysis for passwords stored on the outbound side of the authentication request and prioritize the threats identified.
• Determine what, if any, protection over and above file system protection is effective against those threats on the outbound side of the authentication.
• Incorporate those recommendations into the standard.

Implementation of these suggestions would significantly streamline the assessment process, improve confidence of users, and ultimately improve the level of security achieved in the field.  I call on the PCI Security Council to take this proposal under consideration for their next revision.

Same deal if you are in the San Francisco bay area the week of the 25th.  I haven’t received travel approval for this trip yet but one or two more customer visits will lock it in.

What can you expect on these visits?  In one recent case a customer had locked down most of their channels but through a misunderstanding of CHLAUTH and MCAUSER had inadvertently left the entire network open to anonymous remote admin access.  It took us less than 5 minutes to discover that.  Since they had thought the security implementation had been completed, there were no plans to invest any more time or money in remediation.  They’d moved on to other things.  Once we had a chance to chat, they realized there was more work to do and are busy closing up the remaining holes we discovered.

In another case I worked with a customer to show how they could simplify their architecture by consolidating overlapping clusters, removing unnecessary alias queues and using generic authorization profiles.  This took a bit longer than 5 minutes.  About 30, if I remember correctly.  However, they are now spending much less time on administration and troubleshooting cluster issues.

Not in New York City or San Francisco?  As a product manager, I’m happy to come by for a site visit in the US or Europe.  If I can arrange to meet with one or two other customers while I’m in the area, I can usually get travel approval.

Contact me at…

Voice: 720-395-6997
email: t.rob.wyatt@us.ibm.com

Thanks for accepting the LinkedIn request. I am looking for online websphere MQ and Message broker training for beginners. If you are providing training in these technologies, could you please let me know  the details.

Also, Could you please share your tutorials and videos on these technologies?

Thanks a lot.

Most of my authored content is WMQ security and doesn’t directly fit the request.  However, there are many free, online information sources for new administrators, developers and users of WebSphere MQ.  Below is a quick-and-dirty list which I’ll probably clean up and organize at some point but will get you started.

• WebSphere MQ on developerWorks
• The IBM Messaging Community
• The MQ Dev Community
  • Especially the “Participate” page in the MQ Dev Community
• WebSphere MQ’s Facebook page
• WebSphere Connectivity Scenarios Infocenter
  (There’s not much here today but the intent is to grow the content over time.)
• The WebSphere MQ tag on Stack Overflow
• WebSphere MQ library search at IBM
• MQ On TV You Tube channel
• WebSphere Education videos
• My WMQ Security video
• My “Zero to Hello World” video

Got anything to add?  Put it in the comments.  Just remember, this is about WMQ beginner education resources so even though your admin tool may be the best thing since sliced bread I’ll remove advertising links.

(Side note: What was the best thing before sliced bread?  Jimmy Carr says it was “really massive sandwiches.”)

No, not *that* kind of tuning!

I’ve been keeping a running blog post of questions and answers from the WebSphere MQ Admin seminar this week in Amsterdam, but the more I researched this topic, the longer the answer became.  Eventually, it spilled over into its own post.

It turns out that the behavior and controls have changed from version to version and even across Fix Packs.  That means after applying a fix Pack or upgrading to a new version, things might break.  For a new QMgr it won’t “break” anything but it might not behave as expected if you set the parameters based on the old version.  Here then is the story of the many facets of error log size tuning.

Q: I need to increase the size of my MQ error logs.  Where are the error log file sizes specified?

A: The log file sizes can be tuned using the MAXMQERRORLOGSIZE environment variable, the qm.ini file and/or the QMgr attributes in the WebSphere MQ Explorer Extended Attributes panel.  If the environment variable is changed, the queue manager and local applications must be restarted to pick up the change.  If the ini file is changed, the queue manager (but not the local applications) must be restarted to pick up the change, after which the value will be visible in the WebSphere MQ Explorer Extended Attributes panel.  If the size is adjusted using by altering the queue manager attribute in WMQ Explorer, it takes effect immediately and the value is saved to the ini file by the queue manger. Note that there is no equivalent MQSC command for this attribute.  It is available as PCF and WMQ Explorer exposes it in the Extended Attributes panel.  Other PCF-based tools may or may not expose this attribute.

The features regarding error log size differ slightly by version so here are links and descriptions for each.

As of v5.3, the MQMAXERRORLOGSIZE environment variable allowed for increase of the size of the global error logs and the queue manager’s error logs.  The problem with this was that any process for which the environment variable was not set would truncate the error logs to 256k.  Additionally, on iSeries you needed to delete the existing files in order to make the change as described here.

As of V6.0, an attribute was placed in the qm.ini file that eliminated the problem of having to put the environment variable everywhere.  But the ini file setting only affected the queue manager’s logs and was not available on iSeries. If you wanted to change the global error log sizes, you still needed the environment variable.

As of Fix Pack 6.0.2.4, the the MQMAXERRORLOGSIZE  variable was still available on all distributed platforms, including iSeries, but in the first release and the early fix packs of V6.0 MQMAXERRORLOGSIZE affected only the global error logs.  If the enviroment variable was used but the qm.ini file did not contain an entry, the queue manager’s log file size remained at 256k.  This behavior was changed in Fix Pack 6.0.2.4 so that in the absence of an ini file setting, the environment variable applied to the global and the queue manager error log files.  If both the MQMAXERRORLOGSIZE  environment variable and the qm.ini’s ErrorLogSize and are set, the qm.ini setting takes precedence for that queue manager’s error logs.    The Infocenter page is here.  A Technote that further explains all this is here.

As of V7.0, the MQMAXERRORLOGSIZE variable is completely absent from the Infocenter, however it is mentioned in a Technote as being applicable to that version.  The qm.ini file setting is documented however, and the Infocenter page describing how to change the qm.ini file or use WMQ Explorer to update the setting is here.

As of V7.1, both the environment variable and the qm.ini file setting are documented.  More importantly, the default size of the logs was changed from 256k to 2MB which could eliminate the need for tuning if you had been using a value equal to or smaller than that size.  The Infocenter doesn’t specify that this applies to only QMgr logs or only global logs so it applies to both.  I haven’t confirmed this yet so if you notice it isn’t acting this way, please let me know.  There is a small possibility that the new defaults apply only to QMgr error logs and the Infocenter is wrong.

In the event you have iSeries and would still like to tune the error log sizes above 4MB, the Infocenter page does not mention WebSphere MQ Explorer, nor does it mention any requirement to delete the existing files before changing the setting.  However, the Technote mentioned previously is marked as applicable to V7.1 so plan on deleting the error log files if you change this value on V7.1.

On other distributed platforms, the option to use WMQ Explorer is offered and you can choose that or to update the qm.ini file.  The Infocenter page for that option is here.

Interestingly, the MQMAXERRORLOGSIZE environment variable is mentioned in the Infocenter for V7.1 as applying only to the queue manager files.  The global error log files are not addressed.  Googling for “mq errorlogsize 7.1 technote” doesn’t turn up any hits.  If the Infocenter had not explicitly specified “queue manager error logs” I would have assumed the variable retained the old behavior of applying to the global error logs as well.  If you have a V7.1 setup, I’d be curious what behavior you are seeing with regard to this variable and the global error logs.

As of V7.5, the Infocenter has the same content as for V7.1.  However, since this is the version I have on my laptop and can test easily, I tried setting MQMAXERRORLOGSIZE as a global system variable, rebooted and generated massive quantities of global and QMgr-specific errors.  In all cases, the logs rotated at 2MB.  In instances like this I always assume I’ve got a typo or other PEBCAK error so I’ll try again tomorrow and keep this post updated as I go.

My conclusion from all this is that if error log size is important to you (for example, your QMgr tends to roll them over before you can diagnose an error) then it is worth verifying that your setting is appropriate, especially if you have changed versions or are running a large network with many different versions.  Alternatively, you may wish to capture the AMQERR03.LOG files whenever the timestamp changes.  In windows you may wish to look at Microsoft’s Process Monitor utility and in Linux the iNotify facility looks like it would do the trick.  I’ll look at these at some point when I get a free afternoon and post results, but if you try them before then please let me know how it goes.

Welcome to Amsterdam and thanks for attending!  I will continue to update it throughout the week rather than posting once per day, so feel free to bookmark the page and refresh from time to time.  Please also feel free to contact me directly and I’ll either help you out or find someone who can.  Looking for photos?  On to the Q&A…

Q: You mentioned “sack uber flow” or something like that.

A: Sorry for the accent.  That was supposed to be “Stack Overflow.”  Look up old WebSphere MQ questions here, or enter your own.  If you have an account there, remember to accept answers to your posts when you get a good one, and upvote high quality answers to any questions.

Q: We have an exit on z/OS that takes the value from MCAUSER and puts it into the MQMD.MsgID.  Is that available on UNIX systems?

A: You are in luck!  Please see Secure Messaging Scenarios with WebSphere MQ.  Jørgen Pedersen wrote that exit as part of the Redbooks residency that produced this book.

Q: What if my WebSphere MQ administrator disables my AMS security policies, does some nefarious stuff, then changes them back?

A: Yes, it is true that the MQ Admin has the ability to do many nasty things on the QMgr.  If an application accepts commands by listening to a queue, the MQ admin is effectively an administrator of that application.  Except where AMS is involved and the QMgr is V7.5.  As of V7.5, the queue manager will not let the administrator GET or PUT messages on the policy queue using the message API.  If configuration events are enabled, use of the policy management commands or WebSphere MQ Explorer to edit or delete the policies results in event messages.  As AJ noted in class, converting event queues to topics prevents the administrator from intercepting the publications.

Webcast on WebSphere MQ for z/OS Debugging Abends for Beginners – 11 December, 2012 at 11am EST, 4pm UTC.  Register here.

Q: What is this “RFE Community” of which you speak?

A: The Request For Enhancement Community is where you can submit your requirements for IBM products, including the WebSphere MQ family.  Unlike the previous system, the RFE submissions are public and you can vote and comment on the ones you like, or enter your own.  Because the new system is public, there is much more transparency as to what features are most or least in demand.

Already in the first day we’ve run into a few things that are good candidates for an RFE.  Several people mentioned that the Infocenter reorganization makes it hard to find things such as the content that used to be in the Quick Beginnings manuals.  Someone else mentioned Windows Power Shell was not supported in more recent MQ versions.  If there is an RFE for the feature you want, vote on it.  If not, submit a new one.

Q: Where are the communities mentioned in the session?

A: MQSeries.net is easy to find – go to http://mqseries.net.  If you prefer a list server, just send an email with the words “subscribe mqseries” in the body to LISTSERV@LISTSERV.MEDUNIWIEN.AC.AT or use the web interface.  The list server is hosted by the University of Vienna and has been in operation since the earliest days of WebSphere MQ.  Accordingly, it is known in the community as “the Vienna MQSeries list.”

Q: Glen mentioned the Quick Beginnings Guides but I can’t find them in the Infocenter.

A: As of v7.1, these were broken up into 9 sections and redistributed to the topics that contain similar information.  For each of the platforms there’s an index page with links to the 9 sections where the content was relocated.  If you enter “Quick Beginnings” into the Infocenter search box, it pulls up these pages.

Q: I need to increase the size of my MQ error logs.  Where are the error log file sizes specified?

A: This answer grew too long so I moved it to its own post.

I have been trying to get ACF2 access rules in place to secure Pub Sub Topics but I cannot get it to work as documented. Based on the documentation, the call should be made to hlq.SUBSCRIBE.topicname and hlq.PUBLISH.topicname. I currently set the rules to prevent any access to the Topic EPUB with the following ACF2 entries:

SUBSCRIBE.SYSTEM.BASE.TOPIC UID(*) ALLOW
SUBSCRIBE.EPUB UID(*) PREVENT

Based on the above rules, I should not be able to run a process to subscribe to EPUB, but my process ran successfully. The SUBSCRIBE.EPUB UID(*) PREVENT entry should have given me a security violation (MQ 2035). When I look at the Master region initialization, I do see the following message:

20.49.59 STC04131 CSQH024I +MD9T CSQHINIT TOPIC security switch set ON,
356 profile ‘MD9T.NO.TOPIC.CHECKS’ not found

I am not sure, but it may be that either I am missing a profile or that ACF2 is not making the check for some reason.

My response to the email is pasted in below:

I think the problem here is that Pub/Sub authorizations don’t quite work the way you are expecting.  With profiles for queues, the most specific profile is resolved and a single check is made.  So if you have a profiles like this…

setmqaut -n ‘**’ +put +get +inq
setmqaut n ‘SYSTEM.**’ +inq
setmqaut -n SYSTEM.DEFAULT.LOCAL.QUEUE +put +get +inq

…you can open MY.QUEUE for put and get (it matches ‘**’), are restricted from opening most SYSTEM.* queues for anything but inquire, but CAN open SYSTEM.DEFAULT.LOCAL.QUEUE because its profile is more specific than the other two which also match.

With topics it works a bit different.  Suppose you have the following topic hierarchy:

Price/
Price/Fruit/
Price/Fruit/Apples
Price/Fruit/Oranges
Price/Vegetables/
Price/Vegetables/Kale
Price/Vegetables/Broccoli

Anyone authorized to publish on Price/Fruit is able to publish on Price/Fruit/Apples and Price/Fruit/Oranges because these are subtopics of Price/Fruit.  But they are NOT authorized to publish on Price/Vegetables based on the authorization at Price/Fruit.

Anyone that is allowed to publish on Price/ can by definition publish on any more specific topic, including Price/Fruit/ and Price/Vegetables.  This is because they are subtopics of Price/.

So the question is, what happens when two profiles match the same topic string? The closest analogy to your configuration is that we allow publication on Price/ but prevent it on Price/Fruit.

The answer lies in how the checks occur.  Suppose someone tries to publish or subscribe on Price/Fruit/Apples and there’s no topic definition at that level. The result is that the operation is by default denied at that level, and resolution restarts at the next highest level.  MQ finds the topic object at Price/Fruit and sees that it denies access. But resolution does not stop here because access at a higher level includes access at lower levels, therefore MQ needs to know if access is granted further up, so resolution starts again at the next higher level.

Checks continue up the topic hierarchy until either a) a topic is found at which access is granted; or b) MQ reaches SYSTEM.BASE.TOPIC without finding an access grant, at which point access is denied.  If you compare this to authorizing queues, it is kind of opposite – with topics, the *least* specific profile takes precedence.

Why is this? Well, for starters it is kind of intuitive that if you can publish on a topic, you should be able to dynamically invent subtopics within that topic.  But leaving that aside for a moment, imagine that in my hierarchy an app successfully subscribed at Price/Fruit but did not have access to Price/Fruit/Apples.  In order to enforce the restriction lower in the hierarchy, WMQ would be forced to perform an authorization check on EVERY message rather than once at the subscribe API call. That would be a performance nightmare.  Also, messages for Price/Fruit would be OK but the first message that showed up for Price/Fruit/Apples would cause a fatal authorization error. That would mean that an app that was in Production for years might fail suddenly when someone published to a new subtopic.  That would also be a nightmare.

In your case, you need to remove the authorization at SYSTEM.BASE.TOPIC.

It’s always an MQ problem

One recurring theme in the MQ community is that all problems are MQ problems.  Never mind that they almost always turn out to be application, network, firewall, SAN, account maintenance, resource constraints, human error or even sabotage, it’s an MQ problem.  If all the problem tickets have “MQ problem” in the title and are all assigned to the MQ team, the team and MQ itself can get a bad reputation within the organization.  I’ve been working with MQ since the mid 90′s and it has always been thus.  The complaint was raised again today.

I used to try and correct the record and language used for EVERY incident. This worked for a while but then with a lot of turnover, we are back to where we were a couple of years ago. I have had very little success trying to re-educate the current development staff.

Question, how do you deal with an organizational and cultural issue like this?

By the time I started an MQ admin team at my former employer, MQ had a terrible reputation.  In fact, the only reason I was assigned to it full time was that there were frequent outages and management decided it needed a lot of care and feeding.  One of the developers had become the resident MQ expert and after the first app went into production he provided the MQ support.  After two and then three apps were in production he spent as much time supporting MQ as writing code.  Since I wasn’t qualified to write C code (I was a COBOL programmer then) and since everyone knew a trained monkey could watch over MQ, he got to go back and coding full time and I was given 4 queue managers and a banana.  I quickly discovered that every problem was an MQ problem.  After a while they weren’t just MQ problems, they were T.Rob problems since my name was on every trouble ticket.  I began to think taking on the MQ admin role would ruin my career.

There were some genuine problems with channel stability that I solved fairly quickly by consolidating all the QMgr installations to the latest version and applying fixes.  I also standardized the installations and added things like auto-start on server reboot.  It didn’t take long before the genuine MQ problems were rare exceptions but this didn’t seem to help my reputation.  I’d figure out the non-MQ root of a problem and assign it to the right person, often describing the exact fix.  “You see, there’s this thing called a poison message…”  Or “don’t use the physical IP address on the connection, use the virtual IP address.”  Despite my best work, MQ was still in the title of all the trouble tickets, I was doing all the grunt work and all my management saw was “MQ,” “T.Rob,” “Outages”.  I had tried mounting direct challenges to this perception a couple of times, showing documented cases where root causes were traced back to other systems.  Despite all the chest-beating, management remained unimpressed.

I pondered this over a banana one day and came up with a new approach.  I would create systemic incentives for the application teams to avoid attributing all problems to MQ and positive incentives to avoid the problems in the first place.  Since by this time MQ was blamed for everything, MQ had visibility at fairly high levels.  The spotlight is good, it’s just the spin that is bad.  I started working on the stick phase of my plan.

I began to personally participate in every “MQ problem” that was reported.  Because of the high visibility, things were escalated fast.  I always made sure to provide near-continuous status reporting and included at least one tier of management above whoever it had already been escalated to.  Rather than holding the problem until resolution I would start with MQ and work my way back outward, transferring the ticket as soon as I had documented that MQ was in the clear and had enough additional information to indicate the next logical diagnostic owner.

Since MQ was usually eliminated very quickly as the root cause, most of the problem reporting focused on what was now an app issue, network issue, firewall issue, etc.  Very quickly, all involved would hear “well, we now know it’s not an MQ issue but indicators point to…” and whoever thought escalating to light a fire under me soon found themselves the focus of all the attention.  Not that I pointed fingers, I just presented diagnostics.  I pointed a giant spotlight on the problem and followed the trail confident the trail would usually lead somewhere else.   During the handoff I made sure to attribute the likely source to an app or a component, or even to an administrative action, but never to a person.  Often the root cause was in the system where the problem was discovered and first reported.  In that case I’d give them a free pass or two and then gently suggest that “I helped you resolve these and didn’t point fingers, how about you don’t call these MQ problems when you write them up?”  If that didn’t work, I’d be a bit more public in resolving things quickly as “not an MQ problem.”

Of course, if it actually were an MQ problem, I’d take full responsibility.  I could afford to take this position because by then it hardly ever was MQ’s fault.  The fact that I embraced and took ownership of the things that actually were MQ gave me more credibility when I said something wasn’t MQ.  I also published shop best practices for coding MQ apps and offered lunch-n-learn and other training for project teams.  If someone persisted in blaming WMQ despite all my other efforts, I might point out in my rebuttal that “not only was this not an MQ problem, but the internal MQ Admin web site has docs describing how to avoid this exact problem and we cover it once a quarter in the lunch-n-learn.”

Over time this helped rehabilitate MQ’s reputation as well as my own.  Management now appreciated that doing a good job with MQ perhaps required more than just a trained monkey and the team gained a few top-notch members.  The stick had worked well.  The team was finally staffed for 24×7 support, outages were at their lowest levels ever and we provided this level of service while managing more QMgrs per person than ever before.  In the good graces of management and our application team customers, we had some latitude and a bit of clout.  Now it was time to implement the carrot phase of the plan.

When an app labeled a problem as “MDB not receiving messages” instead of “MQ not delivering messages” and then worked with my team to fix it, I was their best friend.  The outage duration would be much shorter due to the lack of friction and teams claiming ownership of tickets rather than reassigning them.  Afterward, I’d sign off on the problem report saying that the MQ team had collaborated on the solution and that I’d been assured as of the next change cycle it would be permanently fixed.  Of the apps using MQ, those who fought with us had the majority of outages and those who worked closely with us were very stable.  This did not pass unnoticed.  Because of that, earning our endorsement on the trouble ticket improved management’s confidence in the solution and generally diminished negative consequences for the app team.

This sounds like a lot of work and if I’d had to practice it all the time it would have been.  But the point was to build positive and negative incentives into the system to guide people to do the right thing and reward them when they did.  It became risky to label something incorrectly as an MQ problem but politically rewarding to collaborate on solutions.  Problems incorrectly attributed to MQ might end up looking like the app team were trying to shift blame and/or had ignored the training provided.  But state the problem in neutral terms and engage the MQ team early and we’d help you track down the root cause and resolve the outage.  If we weren’t busy, we’d even help you with things that obviously weren’t MQ related since by this time we had working relationships with the network team, firewall team, DNS team, sysops, accounts management and every other discipline that MQ depends on and had become valuable facilitators.

The MQ team earned a rep as the problem solvers rather than the problem source, and that meant something.  It also gave us political capital to enforce best practices and shop standards against an organization over which we had no formal authority.  If the project manager said something was going to Production, the division my team was in had no veto power.  But if I discovered an app wasn’t printing linked exceptions or had some other defect my team considered sev-one, there was a good chance I could block the production implementation even though my boss couldn’t.  No longer content to fend off bad press and aspire to the level of mere neutrality, we used our position as middleware admins to recast ourselves as the A-Team of problem resolution.

Once we established the reputations of MQ and the team, maintaining that position required a lot less work.  In fact, the app teams were some of our best ambassadors.  On several occasions I overheard someone in an adjoining cube say something like “I think there’s a problem with MQ, my app isn’t getting any messages” and another developer would respond “It’s probably a poison message, did you read the poison message guidelines yet?”  Sometimes problems reappeared.  At one point we began to see poison messages after a two or three year remission.  Rather than blaming MQ, management encouraged the responsible app team to consult with us.

As middleware administrators, we often lament that our product gets blamed for everything due to, well, being in the middle.  We protect ourselves and the systems we manage from reputational attack by building walls to deflect attribution and blame.  We dream of the day when people stop dumping their problems on the MQ team.

What we often overlook to our detriment is that being in the middle gives us an unparalleled vantage point from which to observe and interact with all of the mission-critical systems running the company.  If you have some troubleshooting skill and a goal to become the most valuable person in the company’s IT department, you’d have a hard time finding a better position from which to accomplish that then as an MQ administrator.  From this perspective the ideal product to manage would be virtually trouble free yet attract demand from adjacent systems for your services.

You want to assign that problem to me?
Go ahead. Make my day.

Want to change the culture?  I say tear down the walls.  Don’t deflect blame but use it as the raw material from which you refine your reputation as the go-to team for problem solving.  The day people stop dumping their problems on the MQ team is the day the MQ team ceases to be relevant.  But do give them reasons to focus on solutions rather than casting blame.  Provide real value.  Make the other teams look good for the wins and accept responsibility for the losses.  Show the other teams you are much more valuable as an ally than an adversary and don’t hold grudges.  Make it politically expensive to burn you but extremely valuable to engage you in good faith.  Whatever else you do, be happy they keep coming back.  It’s an MQ problem.  It’s always an MQ problem.  It always will be.  That’s a blessing, not a curse.