Question

We are about to start a new project which invovles, at end of process, writing some 5,000 files in various sizes. All files are regular text files and i wonder what is the best way (if someone has experience) to write them.

i was thinking of using file templates (pre-loaded to memory) or direct file streams.

I wonder if someone has experience and can share it with me.
thanks

Answer 1

I would suggest to write a prototype to check in advance if you can meet the performance requirements in the way you would like to realize the project. But don’t forget that harddisks are sometimes hard to evaluate (although their name probably doesn’t come from this fact 🙂 ): They have caches and their performance might differ heavily on background processes, fragmentation, filesystem etc.

A rule of thumb is to reduce the number of file writes. Usually it is the fastest if you first write everything to a memory buffer and then write this buffer to the disk. (A very bad way would be to write char by char.)

Depending on the filesystem it might also be faster to write one big file instead of many small ones, so maybe creating a ZIP archive might be an alternative.

On windows there is the MultiMediaFile IO API (native), which can be faster than standard I/O mechanisms (http://home.roadrunner.com/~jgglatt/tech/mmio.htm) in several cases, even if your content is not “Multimedia”.