I’m planning on deploying a dynamic site that needs certain tasks to be done periodically in the background, let’s say every hour or two. The data that i need to output is strictly depending on the result of those queries.
Now, the problem is that those tasks require a certain amount of time, e.g. 8-10 minutes: so if a user asks for a page, and therefore a certain view gets called, and that task is still in process everything goes bad.
So, here’s the question: what do i need to implement to handle that problem?
I’ve been searching and i found about celery but i haven’t understood how it could be used to fix the above issue and maybe implement a caching system that intervenes when the tasks aren’t finished yet.
A solution can be the following:
The view will serve cached results, when a task is completed this one will update the cached results with the fresh ones.
You can also use celery in this way here, when a cached view is hit and it has to be recalculated, a celery task get detached (the view will return the cached value).
Here there’s an interesting example about a one-at-time celery task:
http://ask.github.com/celery/cookbook/tasks.html