WMQ File Transfer Edition launched

I’ve been haunting the Vienna MQ list server for long enough to have seen the topic of moving files over MQSeries, and later WebSphere MQ, raise it’s ugly head on many occasions.  I say “ugly” because creating a general-purpose program to transport files over a message-oriented protoocol is not trivial.  There are a great many issues to be resolved:

• Does each file become a message?  If so that limits the file size to 100mb.
• If the file can be comprised of multiple messages, they may arrive out of order, have chunks missing, etc.
• Will code page conversion be supported?  How about line ends?
• Assuming for the moment that the endpoints are on compatible filesystems, it is necessary to specify and manage all of the file metadata such as paths, directories, file name collisions, renaming, file creation and deletion, etc.
• Then there are even greater issues moving files across unlike systems where file metadata needs to be converted somehow.
• What provisions will be made for instrumentation such as flow control and scheduling?
• How will the movement of files be tracked?

Obviously this is a lot more complicated than just writing programs to “pipe” a file to a queue and then a queue to a file.  If it were that easy everyone would just use the Q SupportPack.  I know many folks who have implemented solutions written in-house over the years and these have been of varying quality.  For the most part, they were “just enough” solutions fitted to a particular set of requirements.  They were not necessarily of poor quality but in most cases they were so closely tied to the surrounding infrastructure, such as scheduling and monitoring, as to be non-reusable highly customized solutions.

WebSphere MQ File Transfer Edition aims to change all that.  The first version went GA on 5th December and provides a generalized solution that addresses all these issues, and more, on UNIX, z/OS and Windows platforms.  Files are broken into chunks for transmission and then reassembled on the other side.  In addition to allowing files of any arbitrary size to be transferred, this allows you to tune the message sizes to match your other traffic.  For example, if your applications normally move messages of 120k, you probably want to move your files in similar-sized chunks to make use of whatever tuning you have done on your channels, disk buffer sizes, etc.  In addition to the basic necessities of converting between platform semantics, there is a scheduler, code page conversion, line end conversion and a means to audit the file transfers.

As you might expect, I have been looking at the security aspects of the product.  I have a more thorough article due to be published later this month or in early January that will explain all of this in much greater detail but there are a couple of security-related items that I wanted to get out to you right away.  Both of these are covered in the manuals so it is not as though they are not documented, this is just to highlight them so they don’t get missed.

First, run WMQ FTE under a non-privileged user ID.  The installation places files into restricted filesystems, such as /opt on UNIX systems, so the installation tasks require a privileged account.  But to actually run the commands and the file transfer agents does not require any special privileges. [8 Dec 2008 T.Rob: Clarification – you can install to a private location using a non-administrative user ID!]

Next, use the sandbox facility!  Like any other software product, including WMQ itself, you do not want to take the default installation options.  By default, WMQ FTE will read and write from any arbitrary path in the filesystem.  It is entirely valid for instance to specify a file transfer from the root of the file system (C:\ for example on Windows or / on UNIX), with directory recursion and delete after transfer.  The sandbox facility allows you to specify a directory that WMQ FTE will confine its operations to.  I have tried quite hard to construct a path that leads out of the sandbox using various file path traversal tricks and have yet to successfully access anything outside the restricted area.  But you have to enable it for it to be of any use to you because by default it is turned off.  Check the InfoCenter page that describes agent properties for details.

You will also probably want to subscribe to audit logs.  The audit records are published on what is known as the “Coordination Queue Manager”.  This queue manager does not run any WMQ FTE components but does make use of WMQ v7.0 Pub/Sub facilities to publish audit records.  What this means is that if there are no subscribers, the audit messages never get published.  This is a very MQ-ish way to manage these records and quite appropriate.  Now that WMQ has a more robust Pub/Sub I would expect to see native WMQ event messages switched over to publications at some point as well.  (This would resolve the conflict when multiple monitoring agents compete for event messages.)  But it may seem counter-intuitive to find there are no audit messages waiting for you after your new WMQ FTE network has moved a file or two.  To insure the audit messages are published, you will need something to subscribe to them.  This can be as simple as pointing WMQ Explorer at the coordination queue manager, or defining a durable subscription manually or writing something that subscribes to the audit messages and then does something with them such as logging them to a database.  Full instructions are here.

Being that this is the first version of the product, I would expect to see follow-on changes in visible places such as the InfoCenter, the external interface of the commands and even functionality.  For example, I’ll be suggesting that sandboxing be enabled on the initial release.  The install would create an empty directory and the agent’s sandbox parameter would point to it by default.  You could always change it or disable it altogether, but out of the box it would be a lot less dangerous this way.  I have no idea if this will be implemented or how much leeway there is to change the product but I am assuming that the opportunity to make substantive changes will only diminish over time so I’m getting my requests in now.  I would expect customers to have even more leverage at this time than I do so if you are considering a trial of WMQ FTE, now’s the time.