Question

Basically I’m looking for a best way to store partial uploads on server. Files are going to be uploaded chunk by chunk. Chunks may come in parallel and in arbitrary order. I will need to temporarily store them somewhere in somewhat form and then construct complete file out of uploaded chunks. I’ve got some ideas of how to do that, but I wonder if there is some best practice already, or a standard, or RFC of some kind..?

UPDATE:

Upload will happen via HTTP.

Answer 1

This problem has been solved many times before. Use BITS, Torrents, FTP etc

The basics are:

> session initialisation

> send file name, chunk size, number of chunks, hash of each chunk

> send chunk(s) when signalled. Each chunk starts with chunk index number

< receive file name etc

< allocate empty file of correct name and size

< allocate completion file (say name.status) and have one int per chunk. These represent percent complete of that chunk

< signal to start chunks at required indexes

<< As each chunk is being received, save into allocated file at correct location

<< As each chunk is completed confirm hash. If wrong mark as 0% again

< when all chunks are received remove name.status and unlock allocated file

Some systems use large chunks and put file name, size etc at front of all chunks, So chunks are independent and can be received without session initiation