Question

I’m building a Java application that will allow our users to load a list of files and have those files transferred to our server for video encoding. I’ve already built an API for managing the files before and after they’ve been transferred, but I need to decide on a good transfer protocol for actually moving the files.

Right now I’m leaning towards using the Apache Commons Net ( see: http://commons.apache.org/net/ ) package along with FTP to move the files from the client computer to the server. Once there I’ll use secure API calls to move the files to wherever they need to go.

Is this the best route? Is there a better way to reliably transfer large (1 GB+) files? Is there a way to resume a broken download using this methodology? I’d like to avoid traditional HTTP POST requests as they’re unreliable and cannot resume broken uploads.

Thanks!

Answer 1

You didn’t mention if using Amazon S3 is an option for your solution, but they do offer native partial upload support. The basic workflow is:

1. Create an upload-placeholder and hold on to the response key
2. Upload chunks — can be concurrent and retried as necessary
3. Use the response key to combine the chunks into a single file

Their SDK offers built-in file slicing and chunk upload.

Even if S3 is not the final location, you could use S3 as an upload holding pen and download the file at your convenience for permanent storage.