Question

creating a small web application to be deployed on Heroku, where we would like to update an entry in the database whenever a user clicks on an update button. Every day, we would automatically go through and update all the entries which had not been updated by some user in the last 24 hours.

The update function and the web application is written in Django, but because the update involves making a request to another website, it takes a few seconds to process, and I was thinking about making these requests perform in parallel, so multiple users can click on different update buttons at the same time, and have it go through quickly.

We were thinking about using the Scheduler on Heroku for the 24 hour update, but is there a threading application/implementation we could use for this kind of use case? Any tips would be awesome.

Answer 1

Celery is great for scheduled tasks like that. I found a hack to set up workers on Heroku here, although I haven’t used Heroku so you’re on your own there.

The celery usage would look as simple as:

@celery.task
def do_costly_database_update(some_data):
    # do stuff
    pass

# And then in a view...
do_costly_database_update.delay(some_data)

Edit:
Celery also handles recurring tasks like your nightly updates with a config entry like this:

'every-night': {
    'task': 'tasks.do_costly_database_update',
    'schedule': crontab(hour=0, minute=0),
    'args': some_data,
},

All of the Celery docs are really good.