Question

I’m writing an application that deals with very large user-generated input files. The program will copy about 95 percent of the file, effectively duplicating it and switching a few words and values in the copy, and then appending the copy (in chunks) to the original file, such that each block (consisting of between 10 and 50 lines) in the original is followed by the copied and modified block, and then the next original block, and so on. The user-generated input conforms to a certain format, and it is highly unlikely that any line in the original file is longer than 100 characters long.

Which would be the better approach?

1. To use one file pointer and use variables that hold the current position of how much has been read and where to write to, seeking the file pointer back and forth to read and write; or

2. To use multiple file pointers, one for reading and one for writing.

I am mostly concerned with the efficiency of the program, as the input files will reach up to 25,000 lines, each about 50 characters long.

Answer 1

25000 lines * 100 characters = 2.5MB, that’s not really a huge file. The fastest will probably be to read the whole file in memory and write your results to a new file and replace the original with that.