Question

I’m currently writing a program which reads a short string from a (potentially large) number of sources as frequently as every second. I need to be able to write this data to separate files, and I was hoping for some guidance on the most efficient way to do this.

My current implementation queues the data, and flushes it when the queue exceeds a certain size.

Is there a better approach? In C#, are there any IO constructs that are particularly efficient?

EDIT: By large, I believe the “reasonable” maximum would be ~100 data sources, but it could be several hundred in the worst case.

Answer 1

You will have to define “large” to get the best answer. You don’t really need your own queue. The .NET Framework’s BufferedStream is quite efficient

http://msdn.microsoft.com/en-us/library/3dsccbf4.aspx

http://msdn.microsoft.com/en-us/library/system.io.bufferedstream.write.aspx

If “large” does not approach the maximum number of file handles the OS permits, you can simply leave the files open (set sharing as needed if other processes have to access them while you are writing them). That will avoid the overhead of opening each file once per second.

Make sure, whatever buffering approach you use, that you do not buffer more data than you are prepared to lose in the event of a power outage or other system failure.

If you cannot accept data loss, you can immediately write each string to the file (without any buffering) and instead use a disk controller with a write cache and battery backup.

UPDATE

100 data sources is far below the maximum number of open files for any OS that runs .Net. You should be fine just opening the files and leaving them open until you are done with them.

For an interesting read on the limits on the number of handles in Windows see

http://blogs.technet.com/b/markrussinovich/archive/2009/09/29/3283844.aspx