No such thing as a persistent queue!

The widespread usage of the phrase “persistent queue” has a negative impact because people believe that queue attribute actually does something. It’s always worth taking time to stamp out usage of that phrase wherever we find it and I’ll attempt to explain why.

The problem with the term “persistent queue” is that it leads people to believe that the behavior attaches to the queue itself when in fact with WebSphere MQ message persistence is now and always has been an attribute of the individual message. Because of the widespread use of the term “persistent queue” and an incorrect belief about behavior of the queue based on that, some customers find that WMQ doesn’t behave as they expect, often to their detriment.

There are two rules for persistence:

  1. If the quality of message persistence is important, set it explicitly in the application.
  2. Whether or not a message is persistent is always important.

The only time that DEFPSIST should be relied on to actually affect message persistence is the odd case that message persistence needs to be externally settable at run time. The reasoning is that persistence (or not) is a design decision driven by a business requirement. The person responsible for implementing those business requirements is the development team manager who has the opportunity to explicitly choose the correct behavior. When that choice is abdicated and placed in the hands of the WebSphere MQ admins, separated by both distance and time from the actual business requirements, it doesn’t get managed to the original business requirements and that results in a brittle system.

One of my first assignments with IBM was a company who had been experiencing a network-wide WMQ outage for a couple of weeks. We determined that someone had changed DEFPSIST on a queue in a network that was tuned and designed for non-peristent messages. The result was an outage lasting almost 3 week. In another case a customer’s entire Message Broker service infrastructure went down in Production because someone decided that the transmit queue should be persistent.  Immediately after the change all replies to TEMPDYN queues were shunted to the DLQ. Fortunately in that case the DLQ messages immediately told us what the problem was so the outage was of short duration.

The pervasive misunderstanding of the queue’s DEFPSIST attribute continues to cause outages and the external symptom of that misunderstanding is the use of the term “persistent queue.” For instance, during debugging people often report the setting of the queue as if that somehow reflects the persistence off the messages in the queue.  “Where did my messages go?  I have a persistent queue!”  Of course there is no direct cause and effect there. The messages may have come through an alias or a QRemote where DEFPSIST(NO) was set.  Or (hopefully) the application explicitly set the persistence that was required.  I climb on this soap box whenever I hear the phrase “persistent queue” during debugging because DEFPSIST tells me nothing about whether the messages in the queue were actually persistent or not.

Many people who understand DEFPSIST quite well also use the term “persistent queue” simply out of convenience – it has entered the culture and it’s easier than saying “DEFPSIST on the queue is on”. It’s worth asking even those people to not use the term so that a) we can identify the people who need education; and b) so as not to perpetuate the misunderstanding of persistence that leads to brittle systems.

So…there is no such thing as a “persistent queue.” Only queues with DEFPSIST(YES), which contain messages that may or may or may not be persistent.  Practice that discipline in your admin and programming teams and and you will find that the reliability of your systems improves over time.

This entry was posted in WMQ and tagged , , , , , , . Bookmark the permalink.

5 Responses to No such thing as a persistent queue!

  1. Chris Pitman says:

    I understand you are talking about WMQ, but its worth mentioning that other messaging systems have different ways of handling this. For example, Apache Qpid only persists persistent messages in a persistent queue.

    • T.Rob says:

      Good point, Chris! In the context of this blog and WebSphere MQ in general it is the message that persists independent of the setting on the queue. However in the greater world of messaging there are plenty of other approaches taken. I’m only just out of IBM and still a bit xenophobic about messaging transports. 🙂

  2. Laurent says:

    When teeaching the MQ admin course, I always insist on this. If the course could be changed in that direction, that would be a great improvement (and save me some explanations)

  3. T.Rob says:

    Last week an IMPACT attendee begged to differ with my conclusion that there’s no such thing as a persistent queue. “Except for temporary dynamic queues,” he told me with a grin, “they’re pretty much ALL persistent. Once you define ’em, they never go away.” Well, you got me there! This topic actually came up a couple of times and one person suggested there should be a DEFPSIST(REQUIRED) setting on the queue that means “if you do not explicitly set persistence one way or the other, the PUT fails.” I like it! Let’s see if it ends up in the RFE Community. I’d vote on that.

  4. Gerd Diederichs says:

    To expand on this topic, which happens to be a bugbear of mine of long standing as well, the queue’s default persistence is hardly ever invoked. Why? Because the persistence setting of a message can have one of three values: YES, NO, AS_PER_QUEUE_DEFAULT, and guess what, the latter is not the default! If I remmeber correctly, on distributed systems the default is NO and on z/OS it is YES. And – fortunately in my opinion, most programmers will either forget to choose or choose YES or NO, respectively. Which is exactly what we want: People making up their minds about whether they need persistence here or not. What remains as a rather unfortunate side effect is that people look at DEFPSIST of a queue and then expect the messages to go that way. Hence the wail “But my queue is persistent, where are my messages?” – Well, see above for T-Rob’s excellent explanation.

Leave a Reply