Question

SQS expects your application to be idempotent and I’ve got multiple consumers/producers where (even if SQS had a deliver-once mechanism) I will have race conditions creating duplicates and race conditions consuming because my consumers run via cron jobs.

My current plan is to use the Django 1.4 select_for_update which should block other consumers on the same row, doing something like:

reminders = EmailReminder.objects.select_for_update().filter(id=some_id)
if not reminders[0].finished:
    reminder.send()
    reminder.update(finished=datetime.now())
# Delete job.

Are there better ways of dealing with this?

Answer 1

Hook up django-celery to SQS and have it designate a periodic job using celerybeat. Then have celeryd worker(s) running on the same queue anywhere you want. Only one will pick up a job at a time and execute it. No need to introduce DB locking on any level.

As long as your worker is guaranteed to finish its current task before celerybeat fires a new one you will never have a need for a lock. Now if you think there is a chance they may overlap you can introduce states for your notifications where:

1. Any reminder starts in “unsent” state.
2. Your celerybeat sends a request to process unsent emails to the queue.
3. Some worker picks it up and grabs all of them.
4. Immediately the worker transitions all of them to “sending” state.
5. Proceeds to send them one at a time (or in bulk).
6. If sending fails for any, revert their state back to unsent.
7. For all that succeeded transition to sent.

This way if celerybeat fires another job while your original job is not done with the initial batch, you won’t have duplicate emails sent. As an added bonus you can scale the solution and distribute the load.