Question

I have one file which will contain the data for many smaller files inside of it. I would like this file to be able to be written to numerous times at once (to different portions of the file).

What I envision doing is storing many 100MB files in one single file, and storing a reference to where those parts begin and end in the single file.

My question is, what is the best way to append 100MB of bytes to a file, so that I can overwrite that dummy data with the actual data as it is downloaded over the network. This way, if another download request begins at the same time, it can begin adding the next 100MB.

Is there a better way to do this than to just simply write 100MB of 0’s to the file? Is there a method for extending the allocated disk space for a file more efficiently (keeping in mind, it may be getting written to currently for previous bytes of the file)?

Answer 1

On some systems (but probably not all) you can seek past the end of file to create a large file with empty space.

$ perl -e 'open FOO,">/tmp/foo"; seek FOO,1E6,0; print "foo\n"; close FOO'
$ od -c /tmp/foo
0000000  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
*
3641100   f   o   o  \n
3641104