Question

I’m reading in from a CSV file, parsing it, and storing the data, pretty simple.

Right now were using the standard readLine() method to do that, and I’m trying to squeeze some extra efficency out of this processing loop. I don’t know how much they hide behind the scenes, but I assume each call to getLine is a new OS call with all the pain that entails? I don’t want to pay for OS calls on each line of input. I would provide a huge buffer and have it fill the buffer with many lines at once.

However, I only care about full lines. I don’t want to have to handle maintaining partial lines from one buffer read to append to the second buffer read to make a full line, that’s just ugly and annoying.

So, is there a method out there that does this for me? It seems like there almost has to be. Any method which I can instruct to read in x number of lines, or x bytes but don’t output the last partial line, or even an easy way for me to manage the memory buffer so I minimize the amount of code for handling partial strings would be appreciated. I can use Boost, though if there is a method in standard C++ I would prefer that.

Thanks.

Answer 1

It’s very unlikely that you’ll be able to do better than the built-in C++ streams. They’re quite fast. In general, the fastest way to completely read a file is to use a single thread to read the entire file from start to end, especially if the file is contiguous on disk. Furthermore, it’s likely that the disk is much more of a bottleneck during reading than the OS. If you need to improve the performance of your app, I have a few recommendations.

• Use a profiler. If your app is reading a line then parsing it or processing it in some way, it’s possible that the parsing or processing is something that can be optimized. This can be determined in profiling. If parsing or processing takes up substantial CPU resources, then optimization may be worth the effort.
• If you determine that parsing or processing is responsible for a slow application, and that it can’t be easily optimized, consider multiprogramming. If the processing of individual lines does not depend on the results of previous lines being processed, then use multiple threads or CPUs to do the processing.
• Use pipelining if you have to process multiple files. For example, suppose you have four stages in your app: reading, parsing, processing, saving. It may be more efficient to read one file at a time rather then all of them all at once. However, while reading the second file, you can still parse the first one. While reading the third file, you can parse the second file and process the first one, etc. One way to implement this is a staged mult-threaded application design.
• Use RAID to improve disk reads. Certain raid modes can create faster reads and writes.