Question

In SQL 2008+, the following Queue table and Enqueue, Dequeue operations are intended to allow efficient, job queuing within streams among multiple generators and consumers in arbitrary named queues in an optional partially ordered succession. Trivial support for poison message handling through RetryLater(), FailNow(), Reset() (not shown).

CREATE TABLE [dbo].[Queue](
    [ID] BIGINT IDENTITY(1,1) NOT NULL CONSTRAINT [PK_Queue] PRIMARY KEY,
    [PredecessorID] BIGINT NULL,
    [QueueName] NVARCHAR(255) NOT NULL,
    [DataType] NVARCHAR(255) NOT NULL,
    [Data] NVARCHAR(MAX) NOT NULL,
    [RetriesRemaining] INT CONSTRAINT [CHK_RetryCount] CHECK ([RetriesRemaining] IS NULL OR 0 <= [RetriesRemaining]),
    [IsFailed] AS (CASE WHEN [RetriesRemaining] IS NULL OR 0 < [RetriesRemaining] THEN 0 ELSE 1 END) PERSISTED,
    [QueuedOnUTC] DATETIME NOT NULL DEFAULT (GETUTCDATE()),
    [DelayUntilUTC] DATETIME NULL,
    [LastFailedOnUTC] DATETIME NULL,
    [LastFailure] NVARCHAR(MAX)
)

-- Enqueue
INSERT INTO [dbo].[Queue]
(
    [PredecessorID],
    [QueueName],
    [DataType],
    [Data],
    [RetriesRemaining]
)
VALUES
(
    @PredecessorID,
    @QueueName,
    @DataType,
    @Data,
    @RetriesRemaining
)

-- Dequeue
SELECT
    [Dequeued].[ID],
    [Dequeued].[QueueName],
    [Dequeued].[DataType],
    [Dequeued].[Data]
FROM
    [dbo].[Queue] AS [Dequeued] WITH (ROWLOCK, UPDLOCK, READPAST)
LEFT JOIN
    [dbo].[Queue] AS [Predecessor] ON
    [Dequeued].[PredecessorID] = [Predecessor].[ID]
WHERE
    [Dequeued].[IsFailed] = 0
    AND [Dequeued].[QueueName] = @QueueName
    AND ([Dequeued].[DelayUntilUTC] IS NULL OR [Dequeued].[DelayUntilUTC] < GETUTCDATE())
    AND [Predecessor].[ID] IS NULL

Consumer transactions are held open from before Dequeue() through processing application-side and until DELETE, RetryLater(), or FailNow().

• Is this design sound?
• Is this setup deadlock free?
• What negative impact might IDX_Queue_IsFailed_QueueName_DelayUntilUTC or other indexes have on deadlock-freeness?
• What other indexes are beneficial?
• How might table partitioning (by queueName) or other features improve the scalability?

• As conceived, Generators signal to Consumers through an external mechanism that data has been enqueued to avoid tight polling. Instead, (and without SQL ServiceBroker) is there a way to use SQL Locks to efficiently lock a Consumer for which there are no available rows until a generator has written to the named queue?

  CREATE INDEX [IDX_Queue_IsFailed_QueueName_DelayUntilUTC] ON [dbo].[Queue] ([IsFailed], [QueueName], [DelayUntilUTC])

Also, I guess that OrderBy doesn’t matter, as long as we always get one with no predecessor.

Answer 1

My 2c on this:

• first and foremost the dequeue is a read. It must be a write. Use UPDATE WITH OTUPUT.
• don’t use NULL for DelayUntilUTC, use current UTC at insert time. OR expressions may trigger scan, resulting in deadlocks, besides the perf degradation
• the [Queue] clustered index must be as follows: (IsFailed, QueueName, DelayUntilUTC).
• make the primary key non-clustered, or get rid of it altogether, there is not much use of identities in queues.
• get rid of the Predecessor and the LEFT JOIN. You’ll be cursingly troubleshoot deadlocks until retirement with them.

I understand that the last item is probably very hard to palate, but correlation locking with readpast is basically impossible to achieve on a relational table.

I have talked about this subject at length in Using tables as Queues. Queues must be simple in order to scale. Your design is too fancy and is not going to work.