Question

I would to know how I can read a large file on disk and do some work with it efficiently.

I am dealing with this question some time.

I am thinking about it, and I believe a general solution ( not platform tied ) could be using threads and a buffer.

But I am not sure to be the best method.

I have read these entries and both are right.

What is the best way to do a search in a large file?

What's the best way to read and parse a large text file over the network?

So, what’s your advice ?

Thanks

Answer 1

I would do it using a producer-consumer model, with a single producer thread reading portions of the file sequentially and putting them in a buffer for multiple producer threads to process them in parallel.