Question

We come across a scenario where disk space was occupied for empty queues in linux environment.

Our queue manager ended unexpectedly as the file system become full and we need to empty the q file to bring back the queue manager.

But actually we dont have any messages at all in queue. This is showing a particularl queue.

Why the disk space is held here? what is the root cause?

Answer 1

WMQ does not shrink the queue files in real time. For example, you have 100 messages on a queue and you consume the first one. WMQ does not then shrink the file and move all the messages up by one position. If it tried to do that for each message, you’d never be able to get the throughput that you currently see in the product.

What does occur is that WMQ will shrink the queue files at certain points in the processing lifecycle. There is some latency between a queue becoming empty and the file under it shrinking it but this latency is normally so small as to be unnoticeable.

The event you are describing could in theory occur under some very specific conditions however it would be an extremely rare. In fact in the 15 years I’ve been working with WMQ I’ve only ever seen a couple of instances where the latency in shrinking a queue file was even noticeable. I would guess that what is actually going on here is that one of your assumptions or observations is faulty. For example:

Was the queue actually empty?

• The queue was most definitely empty after you blew away the file. How do you know it was empty before you blew away the file?
• If there were non-persistent messages on any queue, the queue will be empty after the QMgr restarts. This is another case where the queue can appear to be empty after the QMgr is restarted but was not at the time of failure.
• If a message is retrieved from a queue under syncpoint, the queue depth decrements but the message is still active in the queue file. If a queue is emptied in a single transaction it retains it’s full depth until the COMMIT occurs. This can make it look like the queue is empty when it is not.

Was it actually the queue file that filled up the file system?

• Log extents can fill the file system, even with circular logs. For example, with a large value for secondary extents log files can expand significantly and then disappear just as quickly.
• FDC files can fill up the file system, depending on how the allocations were made.

Was it even MQ?

• If the QMgr shares filesystem space with other users or apps, transient files can fill up the space.

One of the issues that we see very frequently is that an application will try to put more than 5,000 messages on a queue and receive a QFULL error. The very first thing most people then do is set MAXDEPTH(999999999) to make sure this NEVER happens again. The problem with this is that QFULL is a soft error from which an application can recover but filling up the filesystem is a hard error which can bring down the entire QMgr. Setting MAXDEPTH(999999999) trades a manageable soft error for a fatal error. It is the responsibility of the MQ administrator to make sure that MAXDEPTH and MAXMSGL on the queues are set such that the underlying filesystem does not fill. In most shops additional monitoring is in place on all the filesystems to raise alerts well before they fill.

So to sum up, WMQ does a very good job of shrinking queue files in most cases. In particular, when a queue empties this is a natural point of synchronization at which the file can be shrunk and this usually occurs within seconds of the queue emptying. You have either hit a rare race condition in which the file was not shrunk fast enough or there is something else going on here that is not readily apparent in your initial analysis. In any case, manage MAXDEPTH and MAXMSGL such that no queue can fill up the filesystem and write the code to handle QFULL conditions.