Question

For the first time, I am writing a web service that will call upon external programs to process requests in batch. The front-end will accept file uploads and then place them in a queue. The workers on the backend will take that file, run it through ffmpeg and the rest of my pipeline, and send an email when the process is complete.

I have my backend process working on my computer (Ubuntu 10.04). The question is: should I try to re-create that pipeline using binaries that I’ve compiled from scratch? Or is it okay to use apt when configuring in The Real World?

Not all hosting services uses Ubuntu, and not all give me root access. (I haven’t chosen a host yet.) However, they will let me upload binaries to execute, and many give me shell access with gcc.

Usually this would be a no-brainier and I’d compile it all from scratch. But doing so – not to mention trying to figure out how to create a platform-independent .tar.gz binary – will be quite a task which ultimately doesn’t really help me ship my product.

Do you have any thoughts on the best way to set up my stack so that I’m not tied to a specific hosting provider? Should I try creating my own .deb, which contains Ubuntu’s version of ffmpeg (and other tools) with the configurations I need?

Short of a setup where I manage my own servers/VMs (which may very well be what I have to do), how might I accomplish this?

Answer 1

The question is: should I try to re-create that pipeline using binaries that I’ve compiled from scratch? Or is it okay to use apt when configuring in The Real World?

It is in reverse: it is not okay to deploy unpackaged in The Real World IMHO

and not all give me root access

How would you be deploying a .deb without root access. Chroot jails?

But doing so – not to mention trying to figure out how to create a platform-independent .tar.gz binary – will be quite a task which ultimately doesn’t really help me ship my product.

+1 You answer you own question. Don’t meddle unless you have to.

Do you have any thoughts on the best way to set up my stack so that I’m not tied to a specific hosting provider?

Only depend on wellpackaged standard libs (such as ffmpeg). Otherwise include them in your own deployment. This problem isn’t too hard too solve for 10s of thousand Linux applications over decades now, so it would probably be feasible for you too.

---

Out of the box:

Look at rightscale and other cloud providers/agents that have specialized images/tool chains especially for video encoding.

A ‘regular’ VPS provider (with Xen or Virtuozzo) will not normally be happy with these kinds of workload, but EC2, Rackspace and their lot will be absolutely fine with that.
In general, I wouldn’t believe that a cloud infrastructure provider that doesn’t grant root access will allow for computationally intensive workloads. $0.02