Question

I want to write a big file to the local disk.
I split the big file into many small files and I tried to write it to the disk. But I observed that when I split the files and tried to write, there was a big increase in disk write time.

Also, I copy the files from a disk and write it another computer’s disk(reducer). I observed that there was a big increase in read time as well. Can anybody explain me the reason? I am working with hadoop.

Thanks!

Answer 1

That’s due to the underlying file system and hardware.

There’s overhead for each file in addition to its contents, for example MFT for NTFS(on Windows). So for a single large file the file system could do less bookkeeping.Thus it’s faster.

As arranged by your OS, single big file tends to be written on consecutive sectors of the hard drive where possible, but multiple small files may or may not be written as such. So the resulting increased seek time may account for the increased reading time for many small files.

The efficiency of your OS may also play a big part. For example whether it prefetches file contents, how it makes use of buffer, etc. For many small files it’s more difficult for the OS to use the buffer(and deal with other issues) efficiently.(Under different scenarios it can behave differently.)

EDIT: As for the copy process you mentioned, generally your OS do it in the following steps:

read data from disk->writing data to buffer->read from buffer->write to (possibly another) disk

This is usually done in multiple threads. When dealing with many small files, the OS may fail to coordinate these threads efficiently(Some threads are very busy, while others must wait). For a single large file the OS doesn’t have to deal with these issues.