Question

I am developing a web application using Python, Django and MySql. I have a provision in the application where in a user can upload *.wmv and *.mov files which the system will process and convert it into *.mp4. I was using a single server architecture but as the user base is growing, video conversion consumes 90% of the memory. I am thinking of moving the video conversion / streaming server over cloud using Amazon EC2 so that the conversion takes place on a different server using Redis and Celery. I wanted to know if Redis – Celery supports multi-tier architecture. If anyone has ever successfully achieved this, please let me know the steps. It will be really helpful as I was googling this but couldn’t find any supporting documentation.

Answer 1

If you want to use a Celery queue to process your jobs, you’ll need to:

• Set up rabbitMQ. You can do that on your existing host or on an EC2 instance
• Set up Celery. The Docs are good.
• Set up redis for whatever you need it for. Again, the docs are good
• Configure your existing server to be able to talk to the task queue and redis. Basically, tweak EC2 firewall rules
• Make sure the worker server(s) can see the files. Upload them to S3 is probably the easiest way to shift them back/forth from the cloud
• Change your webserver code to trigger a celery task to do the conversion
• Figure out what to do when the conversion finishes – how do you let the user know it’s done?
• Go.

Don’t worry about “n-tier” etc etc – this is a simple setup that you can get going in a day by following the docs.