Home/ Questions/Q 8460493
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-10T13:37:24+00:00

Perhaps I’m being silly asking the question but I need to wrap my head

  • 0

Perhaps I’m being silly asking the question but I need to wrap my head around the basic concepts before I do further work.

I am processing a few thousand RSS feeds, using multiple Celery worker nodes and a RabbitMQ node as the broker. The URL of each feed is being written as a message in the queue. A worker just reads the URL from the queue and starts processing it. I have to ensure that a single RSS feed does not get processed by two workers at the same time.

The article Ensuring a task is only executed one at a time suggests a Memcahced-based solution for locking the feed when it’s being processed.

But what I’m trying to understand is that why do I need to use Memcached (or something else) to ensure that a message on a RabbitMQ queue not be consumed by multiple workers at the same time. Is there some configuration change in RabbitMQ (or Celery) that I can do to achieve this goal?

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    As noted by others you are mixing apples and oranges.

    Being a celery task and a MQ message.

    You can ensure that a message will be processed by only one worker at the same time.

    eg.

    @task(...)
def my_task(

my_task.apply(1)

    the .apply publishes a message to the message broker you are using (rabbit, redis…).
    Then the message will get routed to a queue and consumed by one worker at time. you dont need locking for this, you have it for free 🙂

    The example on the celery cookbook shows how to prevent two messages like that (my_task.apply(1)) from running at the same time, this is something you need to ensure within the task itself.

    You need something which you can access from all workers of course (memcached, redis …) as they might be running on different machines.

    • 0