Question

I guess my question derives partially from this question: std::ifstream buffer caching. I have a boost serialization archive, and am currently reading out of it with the & operator as suggested on the help. My understanding is this translates to << and >> for input and ouput operations. Now judging from above question, those operators, together with fstream, don’t give optimal performance.

What I need to do is read some few 100k objects and work on them sequentially and then delete them. Right now I literally do

1. Read one object 2. Process it 3. Delete it 4. Repeat.

I am wondering whether I should change this to

1. Read all objects into a queue 2. Process the queue 3. Delete everything

or whether I should implement some buffer for the file and read the binary file into the buffer first?

Or maybe start a second thread and have the second thread read the file with some thread handover?

And partially repeating above question, but does someone have more experience trying to optimize the fstream’s buffer, or maybe even a working example?

Thanks

Answer 1

Changing it doesn’t make sense. By default, it goes through a buffered reader anyway, and this reader anyway reads off the hard drive in 4k chunks. Reading them one by one allows for better memory recycling. The only thing that might make sense is to change the 4k to a different number.