I have a Rails application that does a lot of media encoding. I am handling it via background processes, but I see that CPU gets overloaded and the front end loading times are definitely slower than they should be (or were before the back end part became bigger).
So the problem: Rails app with media encoding features experiences CPU loads and front-end slowed down. Goal – decouple front-end and back-end (media encoding) part.
Question – what is the best approach to split an existing application into two parts (front-end part and back-end part)?
1) Is it a good idea to run two copies of app on two servers and make calls between then POSTing/PUTting information via HTTP (or connecting to a remote db)?
2) Is it a good idea to keep the CPU-intesive part wrapped up in Rails code, or should I aim to strip it off the Rails functionality?
If someone can point to a good guide on running multi-server Rails application, that would be great (search returns questions on multi-server Capistrano deployment, but I need some less specific recipes).
A common approach that works well is to use a work queue like Resque.
Managing the code
For ease of management, keep the processing code and the app “in the app”.
Deploy two app-servers but run resque workers on the processing server.
State changes
If the processing jobs relate to ActiveRecord persisted objects you can poll state from the front-end and periodically update it from the backend during the encoding process.
You might find it useful to use state machine.
Your problems have moved
Now you’re cloud-scale™ 😀
More processing hosts running workers can be added if your queue gets too long. If your front-end is the only web-accessible host you can set up a rack middleware or run rainbows to proxy the processed results through the frontend to the client.
Sounds like an interesting project. Good luck!