Store and Forward

Deep Queue #14 – The Elephant Under the Bed

T.Rob — Fri, 27 Nov 2009 05:14:27 +0000

This episode of The Deep Queue celebrates the first birthday of the podcast with some discussion of the SSL protocol vulnerability that was recently discovered. Although there has been no announcement with regard to WebSphere MQ, I try to put the whole discussion into a larger context and ask if this is really the thing we need to worry about considering what else is going on. More on that in the podcast or transcript.

Links for this episode:

IBM e-notifications service
http://www-01.ibm.com/software/websphere/support/einfo.html

Security Now podcast: http://bit.ly/2sLizB
The episode you want is #223 and it should be online this Thursday, November 19th.

Presentation: New Tricks For Defeating SSL In Practice http://bit.ly/NPtwf
from Black Hat 2009 by Moxie Marlinspike

Tools: SSLSTRIP from thoughtcrime.org http://bit.ly/3mu8QB
Provided by Moxie Marlinspike

In memorium

T.Rob — Thu, 29 Oct 2009 22:44:39 +0000

The PCI community has suffered a great loss with the passing of Dave Taylor recently. Dave had a vision of helping companies achieve not just the letter of PCI compliance, but the spirit of PCI compliance through better understanding and an open dialog amongst practitioners, auditors, users, business leaders and all other stakeholders. That vision and passion became the PCI Knowledgebase, billed as “the largest PCI research community.”

I first met Dave when I was invited to present a webinar on WebSphere MQ security to the PCI Knowledgebase community. By Dave’s standards the webinar was barely attended but as a conference speaker the numbers were about what I’m used to. I presented that webinar twice and have since heard back from many of the attendees that they are now including WebSphere MQ in their PCI assessments and that they are finding – and fixing – configuration issues. This is exactly the kind of thing Dave was trying to achieve and, although he was initially skeptical about the attendance rates, I know he was happy with the results.

The PCI Knowledgebase has pledged to continue in Dave’s absence and carry on his mission. If you are a PCI DSS stakeholder, please stop by the PCI Knowledgebase web site and check it out. It’s a great resource for anyone involved with PCI DSS and your participation is the best way I can think of to honor Dave’s memory.

WMQ free tools updated

T.Rob — Thu, 29 Oct 2009 22:27:07 +0000

Michael Dag at MQSystems has updated his WMQ 1-Page Quick Reference cards. Go to his Links page and look at the top two entries. While you are there, check out the Solutions page where you can find out abour MQ Document and MQ Architect.

Roger Lacroix has updated a few of his free products recently, including MQWhat. You can find these on his Open Source page at Capitalware.

Avoiding insider threat

T.Rob — Tue, 06 Oct 2009 03:43:01 +0000

Passing along this article from Adam Bosnian of Cyber-Ark Software: Practical advice on avoiding the insider threat. The whole article is worth reading but one item stood out:

Best Practice #4: Secure Embedded Application Accounts

Up to 80 percent of system breaches are caused by internal users, including privileged administrators and power users, who accidentally or deliberately damage IT systems or release confidential data assets, according to a recent Cyber-Ark survey.

Many times, the accounts leveraged by these users are the application identities embedded within scripts, configuration files, or an application. The identities are used to log into a target database or system and are often overlooked within a traditional security review. Even if located, the account identities are difficult to monitor and log because they appear to a monitoring system as if the application (not the person using the account) is logging in.

These privileged, application identities are being increasingly scrutinized by internal and external auditors, especially during PCI- and SOX-driven audits, and are becoming one of the key reasons that many organizations fail compliance audits. Therefore, organisations must have effective control of all privileged identities, including application identities, to ensure compliance with audit and regulatory requirements.

Wow, 80% of breaches now originating within the “trusted internal network” and the article also mentions insider theft has doubled between 2007 and 2008. Anybody out there still not treating the internal network as a hostile environment?

Deep Queue #13: Unlucky number thirteen

T.Rob — Fri, 28 Aug 2009 17:19:10 +0000

After a month unplanned hiatus, The Deep Queue is back. This month we are talking about high availability, which is obviously something The Deep Queue lacks. This may not seem like a security topic at first glance but to my way of thinking it is. There is no perfect security so, no matter how much you invest in protection, it’s a good idea to assume there will be a breach at some point. When that happens you need to detect it and recover from it. For that reason, whenever I perform a security assessment, monitoring and recovery are considered. The first segment talks about planning for recovery from an adverse security event.

The second segment is all about the new v7.0.1 of WebSphere MQ! Yes, the long awaited refresh pack is released and you can download it now. In addition, the product manuals are all updated with the new features. The v7.0 release was massive and it seems hard to believe it would be followed so quickly by another release with this much functionality but the folks in Hursley apparently don’t sleep. There’s lots of detail in the podcast and at the links below.

Links for this episode:

IBM WebSphere MQ V7.0 is enhanced with increased availability, security, and governance
IBM United States Software Announcement 209-245
August 25, 2009
http://bit.ly/1anQmy

IBM WebSphere MQ for z/OS V7.0.1 delivers storage and memory improvements and increased availability for queue-sharing groups
IBM United States Software Announcement 209-248
August 25, 2009
http://bit.ly/3bv4pv

developerWorks: Mission:Messaging: Planning for SSL on the WebSphere MQ network
http://bit.ly/3loir

WebSphere MQ – Coming soon to an audit near you!

T.Rob — Sat, 04 Jul 2009 15:34:52 +0000

The June 29 episode of The Deep Queue is finally up! Sorry about the delay, I was on an engagement last week that had me staying over the weekend in Boston to perform a production implementation on Saturday. Although I’ve got a great recording setup at home, I’m afraid I don’t have decent equipment to do the podcast on the road. Instead, I flew my wife up to Boston and we spent Sunday at the aquarium and then went to see Blue Man Group.

The week delay worked out great though, because last week a friend contacted me to tell me his shop needs to remediate for PCI compliance. He has a hundred days to create a segmented MQ network within which to isolate his PCI applications. The time limit is due to having found out about the problems in the course of an audit rather than through independent research or assessment. Since this is likely to be a growing problem, it turned out to be my topic for this month’s episode.

The reason I think this will be a growing problem is that I am among the folks talking with the assessment community about WMQ security, the implementation gaps that are commonly seen and methods for assessment and remediation that are currently available. Hopefully, the participation of the assessment community will result in refining these existing tools and creating best practices for securing MQ in a regulatory compliance context such as PCI.

I’m also excited to be working with some old friends at Evans Resource Group. ERG is building a business around helping assessors get up to speed with WebSphere MQ. They are creating a curriculum and tools and are already working with some of their first clients in this space. Many of the folks at ERG are Reconda alums who I worked with to develop AppWatch so I’m confident they will do a great job. I’ll be working with them next week to help them develop and fine-tune their content and get the reactions of those initial clients.

Lots more about all this in the podcast so please download it or the transcript and let me know your thoughts.

Also, don’t forget to sign up for the webinar I’m giving July 10th at noon Eastern, entitled What You Don’t Know About Middleware Vulnerabilities Will Hurt You. The webinar is structured for assessors and QSAs and includes my 5-Minute WebSphere MQ Assessment.

Links from the podcast:

PCIKnowledgebase.com: http://PCIKnowledgebase.com

Webinar: What You Don’t Know About Middleware Vulnerabilities Will Hurt You
https://www2.gotomeeting.com/register/848961386

Evans Resource Group home page: http://www.evansresourcegroup.com

Evans Resource group free MQ security check:
http://www.evansresourcegroup.com/technologies-6b.html

Prolifics home page: http://www.prolifics.com

Prolifics free MQ Health and Security Check:
http://www.prolifics.com/Collateral/Documents/English-US/service-brochures/Prolifics_WebSphereMQ_HealthSecurityCheck.pdf

Capitalware homepage: http://www.capitalware.biz

Capitalware consulting services: http://www.capitalware.biz/services.html

Primeur homepage: http://www.primeur.com

Primeur Data Secure for WebSphere MQ:
http://www.primeur.com/products/data_security/spazio_data_secure.html#dswmq

New WMQ Channel vulnerability and interim fix announced

T.Rob — Fri, 05 Jun 2009 22:04:20 +0000

The IBM Internet Security Systems XForce team recently announced a buffer overflow vulnerability in WebSphere MQ client channels. According to the release, the vulnerability includes the possibility of remotely executing arbitrary code or “causing the application to crash.” It is not clear whether “application” in this case refers to the channel agent, channel pooling process or something else.

I’ve already fielded some questions on this alert. In particular, the following:

Note: This vulnerability can not be exploited on queue managers secured with security exits or authentication through SSL, unless an attacker has valid authentication credentials or a valid SSL certificate.

First, I think that the words “queue managers” here should be “channels”. A queue manager does not have SSL or security exits, channels do. And Ihave no reason to believe that enabling a security exit or SSL on one channel solves the problem for the entire queue manager so I think the scope is wrong. I’ve sent in a suggestion to fix that but haven’t heard back yet.

The second question I received was about how authentication prevents the exploit and whether it is necessary to apply the interim fix. The SSL handshake must be completed before the channel agent ever sees the connection so any connection rejected by SSL does not get deep enough into the MCA to hit the vulnerability. Similarly, the security exit is invoked fairly early on in the channel negotiation. If an attacker’s connection is rejected by either SSL or a security exit, the vulnerability cannot be exploited.

On the other hand, anyone who can complete the connection can execute the exploit. But is this dangerous and do I need to apply the fix? If the channel has a blank MCAUSER and the command server is running, then a legitimately connected connected user already has the same level of access by gaining administrative access and defining services. There are two conditions in which the fix could be important:

The channel has no SSL or exits and relies on a low-privileged MCAUSER value for security. In this case, anonymous users are allowed connect but will ordinarily not have administrative access.
The channel has SSL and/or exits to authenticate users and a low-privileged MCAUSER that would ordinarily limit the connected user from gaining administrative access and the legitimate users are not trusted. For example, when allowing client connections from outside the enterprise or when in a regulated environment in which application controls must be strictly enforced.

For now the fix is available separately but it will be incorporated into 6.0.2.6 and 7.0.1.0 releases of WebSphere MQ. Additional information may be found at APAR IZ50784.

Deep Queue #11: Security breaches are not news?

T.Rob — Mon, 25 May 2009 15:44:08 +0000

The subject of the UC Berkeley data breach was discussed on the May 15th Security Squad podcast. The thing that struck me was that the breach itself was not the topic of conversation but rather the debate was about whether the breach was in fact newsworthy. If you are not familiar with it, 160,000 Social Security numbers and medical information were stolen in the UC Berkeley data breach. This notion that breaches of the “trusted” internal network are so common that’s possible to discuss with a straight face whether a breach of this magnitude is newsworthy is itself worthy of some discussion.

In this episode of The Deep Queue, I tackle this topic as well as the idea of software “never events”. The term was coined in the medical professions to refer to preventable events with serious or deadly consequences. The kind of events that should never happen such as operating on the wrong body part or wrong person. the National Quality Foundation has developed a list of 28 such events which are used to report and track quality of care across the nation. Bob Charette guested on the CERT Security podcast to campaign for a similar set of events in the software industry. In this episode of The Deep Queue, I propose my own list of WebSphere MQ never events.

Links for this episode:

University of California Berkeley Data Breach
http://datatheft.berkeley.edu/news.shtml
Security Squad, SearchSecurity.com podcast for May 15, 2009
http://itknowledgeexchange.techtarget.com/security-wire-weekly/squad-data-breach-burn-out/

PrivacyRights.org Chronology of Data Breaches
http://www.privacyrights.org/ar/ChronDataBreaches.htm#CP

BankInfoSecurity.com – List of banks reported to have been affected by the Heartland breach tops 600
http://www.bankinfosecurity.com/articles.php?art_id=1200

National Quality Forum – Serious Reportable Events (a.k.a. “Never Events”)
http://www.qualityforum.org/projects/completed/sre/fact-sheet.asp

CERT Security podcast series for May 5, 2009
http://www.cert.org/podcast/

WebSphere MQ Security Heats Up – Blog post with downloadable setmqaut scripts to secure administrative access to WebSphere MQ.
http://t-rob.net/2008/07/08/websphere-mq-security-heats-up/

WANTED DEAD OR ALIVE: WMQ Security exits

T.Rob — Fri, 15 May 2009 21:44:34 +0000

As you know, there are some security functions in WebSphere MQ that require an exit. By now everyone should be familiar with BlockIP2, the well known channel security exit. But there are a couple of other requirements that a channel exit can’t meet. In this post I’ll describe what those are and post some specs for an exit. I’m not qualified to write an exit but I’m hoping someone who is will do so. If ever these exits show up on the Internet, you can bet I’ll be posting links to them from my site and referring people to them in presentations…well for at least as long as IBM doesn’t have a solution that I can point to, anyway.
Cluster security

WebSphere MQ clusters are supposed to reduce the administrative burden in medium and larger networks. They do this by automagically defining channels between any two nodes in the cluster. If you take security out of the equation, then this does indeed save a lot of administrative overhead. roblem is, you don’t get the same granularity that you do with a point-to-point network.

With point-to-point you have the option to put a different MCAUSER on every channel. The granularity of your security model can be per-node or, if you define multiple channels between nodes, per channel. With WMQ clusters you get a single CLUSRCVR channel per cluster. If you want to get more granularity you need…you gueesed it…a security exit that can dynamically set the MCAUSER based on IP address or certificate distinguished name.

“But wait”, I hear you saying, “BlockIP2 does all that.”

True. But now the question is how do you get BlockIP2 into the SCYEXIT field of the CLUSRCVR? Imagine for a moment that your nework contains a mix of Windows, UNIX and z/OS queue managers. When you set the SCYEXIT field on the CLUSRCVR of a UNIX queue manager, it looks something like this:

/var/mqm/exits64/BlockIP2(BlockExit)

Now when the CLUSRCVR definition gets propagated out into the cluster, all the CLUSSDR channels pointing to this QMgr inherit the setting. If they happen to be UNIX queue managers, no problem. Assuming the exit is present in the same location, anyway. But the CLUSSDR channels on the Windows and z/OS queue managers cannot possibly work because the path is specific to UNIX.

To address this problem requires a Channel Auto-Definition (a.k.a. CHAD) exit. The purpose of a CHAD exit is to enable some control over what happens when channels are auto-defined. In the case of WMQ clusters, the CHAD exit is driven when the channel is first defined and then again every time the channel starts up. For our purposes, the interesting functionality would be using the CHAD exit to populate the channel’s SCYEXIT and SCYDATA fields. This way we can leave SCYEXIT blank so that it won’t propagate out into the cluster, but still enjoy the benefit of having a security exit on the CLUSRCVR channel.

My specs for a CHAD exit are thus:

Requirements

The exit must be capable of populating, at a minimum, the SCYEXIT and SCYDATA fields of a channel. MSGEXIT/DATA and SND/RCVEXIT/DATA would be useful as well, although not essential at this point.
The exit must be capable of distinguishing at least between channel types. We probably want to suppress auto-defined SVRCONN and RCVR channels.
The exit parameters must have per-channel granularity. We may want to apply different settings to different CLUSRCVR channels.

Nice-to-have

The exit should allow default settings and string-matching on channel names. Any channel not explicitly matching a named string inherits the defaults.
The exit should allow setting of additional fields: SSLCIPH, SSLCAUTH, SSLPEER, MAXMSGL, MCAUSER, and possibly others. (Comments anyone?)
The exit should allow specification of a file or namelist to store parameters. It needs to support different files per QMgr when there are multiple QMgrs on a server.

Message spoofing

The second exit addresses a problem that is common in appplications that use WebSphere MQ, specifically that they generally do not check the message type before processing the message. The reason that this is a problem is the ability of a message producer to request confirmation on arrival. When COA is requested, the queue manager sends the report message to the queue specified in the Reply-To fields of the original message. The requestor can specify that the generated report message contain a copy of the entire original message. In the case of COA, the report message is enqueued with the authority of the queue manager.

As an attacker, what this means to me is that I do not need authorization to a queue in order to put a message onto it. All I need is a channel to your queue manager. I then specify COA on my malicious message and specify the desired target queue in the Reply-To fields. The queue doesn’t even need to be on the queue manager that I have access to since I can address it to routable network destination.

Of course, IBM’s advice in the Application Programmer’s Manual all along has been to check the Message Type field before processing any message. Typically, it should be either a request, a reply or a datagram. The problem is that not too many applications bothered to do this basic validation. Worse, many JMS shops consider this validation to be vendor-specific. In truth, it is possible to write the validation in portable code but the property itself is specific to WebSphere MQ so it is not often checked. The result is that many applications are vulnerable to this spoofing attack.

With so many applications vulnerable, at this point the best way to address the issue is probably with a message exit. My specifications for a messsage exit are these:

Optionally turn off COA on messages arriving over the channel.
Optionally turn off COD on messages arriving over the channel.
Optionally move MCAUSER to the MQMD.UserID
Optionally move an arbitrary parameterized string to MQMD.UserID
Optionally set MQMD.ReplyToQMgr to the name of the QMgr running the exit.
Optionally set MQMD.ReplyToQMgr to an arbitrary parameterized string.
Optionally set MQMD.ReplyToQueue to an arbitrary parameterized string.
Optionally set MQMD.ReplyToQueue by mapping the destination queue to a parameterized string.
Options determining disposition of messages not addressed to valid queue:
- Move to DLQ (with DLQ header)
- Move to queue specified by parameter
- Reject and force channel to stop
- Discard
Options determining notification method
- Write to exit log file
- Write to event queue
- Write to AMQERR01.LOG
- Combination of the above

Note that a message exit does NOT solve all of the problems with vulnerable applications. For one thing, there is no message exit for WMQ client channels. The implication is that any client connection allows the ability to put COA messages onto any queue.

Summary
As of this writing, I don’t know of any free or commercial CHAD exit for WebSphere MQ. Similarly, I don’t know of any message exits that have the functionality I’m looking for regarding confirmation messages. My intent here is to at least hoist up a straw man for discussion. As I stated up front, I’m not qualified to write the C code myself. But I’d be happy to host it here if someone else wants to write it.

Wrapping up IMPACT 2009

T.Rob — Fri, 08 May 2009 16:47:16 +0000

Well, this is the last day of IMPACT. It’s always lightly attended as many folks take Friday as a travel day. I have one more session this morning though. It’s the WMQ ESE introduction. Overall the WMQ security sessions were well attended. Even the small rooms were large, compared to past conferences, so my feeling is that interest in WMQ security seems to be rising. That’s a Good Thing.

I also met with many different folks in one-on-one meetings, Premium Support Zone and just walking around the hallways. It’s great to talk with you in person and is for me the most valuable part of the conference. Thanks so much for talking the time to stop and chat with me.

My theme for the conference has been “your feedback is essential”. Phil Parry and the rest of the Useability team ran several feedback sessions during the conference, including some for WebSphere MQ. Morag tells me she heard from many of you during the conference. The plan is coming together! Keep up the good work.

I have a day at home this weekend to wash clothes, pack, do some lawn work and then back on the road to Boston. Monday evening when I finally come up for air I plan to work on a few new postings for the blog and start on the next Mission:Messaging column. So stay tuned, more news and content is coming. One of the things I want to do is post a “security feedback reporting kit” with links to the WMQ requirements form. I hope there will be lots of comments adding to that post with more suggestions and letting us know what response you get from the lab.

That’s it for now – another customer meeting in 15 minutes to get to.