Question

I am building a web application that allows a user to upload an image. When the image is uploaded, it needs to be resized to one or more sizes, each of which needs to be sent to Amazon s3 for storage. Metadata and urls for each size of the image are stored in a single database record on the web server. I’m using a message queue to perform the resizing and uploading asynchronously (as there is potential for large images and multiple resizes per request). When the resize/upload task completes, the database record needs to be updated with the url.

My problem is that the worker executing the task will not have access to the database. I was thinking of firing off a http callback from the worker back to the web application after the task is complete with the appropriate information for updating the database record. Are there any other alternatives or reasons I should do this another way?

I’m using python/pylons for the web backend, mysql for the database and celery/amqp for messaging.

Thanks!

Answer 1

It seems that your goal is not to decouple the database from the MQ, but rather from the workers. As such, you can create another queue that receives completion notifications, and have another single worker that picks up the notifications and updates the database appropriately.