Question

I have a queue interface I want to implement in redis. The trick is that each worker can claim an item for N seconds after that it’s presumed the worker has crashed and the item needs to be claimable again. It’s the worker’s responsibility to remove the item when finished. How would you do this in redis? I am using phpredis but that’s kind of irrelevant.

Answer 1

To realize a simple queue in redis that can be used to resubmit crashed jobs I’d try something like this:

• 1 list “up_for_grabs”
• 1 list “being_worked_on”
• auto expiring locks

a worker trying to grab a job would do something like this:

timeout = 3600
#wrap this in a transaction so our cleanup wont kill the task
#Move the job away from the queue so nobody else tries to claim it
job = RPOPLPUSH(up_for_grabs, being_worked_on)
#Set a lock and expire it, the value tells us when that job will time out. This can be arbitrary though
SETEX('lock:' + job, Time.now + timeout, timeout)
#our application logic
do_work(job)

#Remove the finished item from the queue.
LREM being_worked_on -1 job
#Delete the item's lock. If it crashes here, the expire will take care of it
DEL('lock:' + job)

And every now and then, we could just grab our list and check that all jobs that are in there actually have a lock.
If we find any jobs that DON’T have a lock, this means it expired and our worker probably crashed.
In this case we would resubmit.

This would be the pseudo code for that:

loop do
    items = LRANGE(being_worked_on, 0, -1)
    items.each do |job| 
        if !(EXISTS("lock:" + job))
            puts "We found a job that didn't have a lock, resubmitting"
            LREM being_worked_on -1 job
            LPUSH(up_for_grabs, job)
        end
    end
    sleep 60
end