Home/ Questions/Q 4118042
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-20T22:54:35+00:00

Let’s take the following example: We have users which asked to post the article

  • 0

Let’s take the following example:

We have users which asked to post the article with id:5 at 2011-04-19 20:20
So I want to create a task which will change the status of the article at 2011-04-19 20:20.

I came up with the following ideas:

  1. Implement Celery and use RabbitQM or Django-kombu: using countdown and eta: http://docs.celeryproject.org/en/v2.2.5/userguide/executing.html#eta-and-countdown
  2. Create a django management task, which will check if a article needs to be published. run this task every minute with a cron job.
  3. Create a small python application with an endless for loop, which checks every second or every 10 seconds. I will put all the articles of the current 10 minutes in memory to do faster checks if there are none it will sleep for 10 minutes.

What do you suggest in this situation? What is best for performance and scaling. Let’s say every second about 100k tasks are executed.

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    My recommendation would be to keep it simple to start, using a management command kicked off by CRON every minute, and when that outgrows your need switch to the distributed job approach. If you separate your code correctly, this shouldn’t be that large of a change.

    If you are going to be doing 100k tasks from the start, I would pick option #1 since you will be able to use celery to distribute the load over many servers. If you pick a normal cronjob setup you will need to run all of the tasks on the same server, which doesn’t scale very well. Setting up RabbitMQ and maintaining it is a lot more involved then setting up a cronjob, so put that off as long as you can.

    For option 2: Django-Extentions also has a cronjob like system implemented as management commands, so you wouldn’t have to reinvent the wheel, and Django-Extentions has lots of other great tools, which you might want to use anyway.

    http://packages.python.org/django-extensions/jobs_scheduling.html

    If you go for #3 make sure you use something to keep the daemon running, If it crashes you will need to start it backup automatically. http://supervisord.org is a good choice.

    • 0