Question

I am building something like a delay-line: one process RPUSHes objects into a list, another LPOPs them out in the same order.

The trick is that objects should only be popped from the list one hour after they have been added. I could use a time-stamp for that. The delay is the same for all items and never changes.

Now how to I implement the pop in a concurrency-friendly way (so that it still works when several workers access that list)? I could take out an item, check the timestamp and put it back into the list if it’s still too early. But if several workers do that simultaneously, it may mess up the order of items. I could check the first item and only pop it if it’s due. But another worker might have popped it then so I pop the wrong one.

Should I use the WATCH command? How? Should I use sorted sets instead of a list? Help appreciated!

Answer 1

Putting the item back in the list after checking it won’t mess up the ordering enough to matter – by having any sort of concurrency you accept that the order won’t be strictly defined. However, that approach won’t be particularly efficient as you can’t check the timestamp without some processing of the item.

With the scripting build you could implement a conditional pop command for sorted sets, though I don’t think that is the easiest solution.

There are a couple of ways to implement basic scheduling using multiple lists:

• Add tasks to a sorted set and have a single worker responsible for moving them from the sorted set to a list that multiple workers access with pop.
• Have a list for each minute, and have the workers read from past lists. You can minimize the number of lists checked by using another key that tracks the oldest non-empty queue.