Question

So I am processing a 200 mb txt file and I have to read each row in the file update one or two columns and then save the same. What is the best way to achieve the same?

I was thinking of lading into a datatable but holding that big of a file in memory is a big pain.

I realise I should do it in batches but what is the best way to achieve the same?

I dont think I want to load into a dB first cos I cant do a mass update anyways. i Have to do a line by line read there too.

Just as an update my files basically have columns in any order and I need to update two or more columns all the time.

Thanks.

Answer 1

Read a line, parse it, and write fields into a temp file. When all the lines are done, delete the original file and rename the temp file.