Question

I am currently developing a service using Java and Spring MVC where users can upload video files. These can later be retrieved through our API.

When the video file is loaded through our API we just send the file completely, as a download, there is no streaming or multiparting. I understand this only works until you reach a certain file size. Does anybody have a recommendation about what the max file size should be? I.e. at which file size should we start using a video streaming service instead?

Answer 1

That is a fairly complex question, actually, as there are multiple factors to be considered. I will list the pros and the cons of each approach, and eventually will provide an alternative solution.

Downloading/Uploading the whole file at once:

->Pros:
Easy to program

->Cons:
As soon as a user tries to upload a two gigabyte file, this file will be in the memory of your server before being saved to your filesystem or wherever you save it. You can imagine that with 100 users doing that, your server will crash. You can limit the size of a request on a servlet container level, but then you will be limiting your users. Another way would be to limit it on an application level as well, but still, you are limiting your users. A couple of years ago, a default size of an upload was 2097152 (2 megs) for a tomcat. Not sure how much it is now, but even if you bump it to 10mb or 100mb, then you run in the problem I described: when multiple users try to upload big files, you have them in memory.

Downloading/uploading the file using streaming

->Pros:
Streaming will not require all your content to be present in memory before saving it. It is a more elegant approach and definitely better in all regards than sending everything. Also, you probably will bypass most problems in corporate environments like sending large files, firewalls, servlet container limits, etc.

Also, if you implement streaming with progress bar, users sending large files to your system wouldn’t think it crashed, which will significantly better your user experience.

->Cons:
Not really much, but slightly harder to implement. With a library like commons’ IOUtils you should not have any problem in implementing a streaming solution.

As you see, it is in most cases better to stream the contents of your file, regardless of its size, but if you still want to use your whole file solution, you could use a limit of 10MB or something in the area. It depends on what size you expect your videos to be.

One thing to note is that if you want to also use streaming to allow the user to see the video contents, you would unneccessarily burden your servlet container with stuff it is not supposed to do: stream videos. Servlet containers are for answering http requests, and by design are made with pools that reuse threads expecting a short lived http request. It might become evident at a certain point that a http servlet container is just not suitable for streaming video AND serving http requests at the same time. A possible solution could be your users to upload your files to a video server using your api, and then use the video server, which might even be on a different location, to stream the videos back to the users. You can take a look at this one:http://www.red5.org/

What you can get out of this setup is:
->You alleviate the load on your http server and you are using it for what it is meant to do: serve http requests
->You reduce application complexity, as stuff with streaming, playback, etc is handled not by your application but by the video server.