Question

I’m in the middle of technically speccing a file uploading API that will be dealing with a lot of large (250MB+) files.

What I want to avoid as much as possible is the web server eating up a whole load of memory during the file upload, which I know to some degree is possible to avoid.

Also, I can’t use Flash or Silverlight to split the files, which would probably significantly decrease memory usage.

I understand that PHP is largely dependent on its host web server software for upload performance. With this in mind, is Apache the best one for the job or should I consider nginx or others?

If PHP itself is the largest bottleneck, then would it be worthwhile considering some sort of hybrid solution? And if so, what language would it be?

Answer 1

Despite what Daniel Ribeiro says, I’d suggest you keep these uploads well away from Apache. You need an event based server (Apache’s implementation is good but IMHO not as polished as nginx / lighttpd). The problem is only indirectly going to be memory – it’s about the number of clients and bandwidth. Splitting the files isn’t really going to help.

OTOH there’s nothing to stop you using nginx to handle the uploads and apache to serve up the rest of the site.

You’ll get a lot of benefit out of tuning the TCP/IP stack to handle LFNs (large congestion window, tcp window scaling).

I understand that PHP is largely dependent on its host web server software for upload performance

The performance of uploads is nothing to do with PHP unless you try to implement the upload using raw socket / web socket with PHP at the back end. Indeed it would make sense to invoke a (forking) PHP interpreter from nginx to process the file – since this only is started when the file is uploaded