I have very less experience of working on large files and constraints about memory.
Please note that I am using Java as programming language.
I have to find records in file size of 15GB and then write result to another file. Basically it is a search range function implementation. My goal is to write the records that lie in my range to result file separated by new line. That is each record should come on separate line. And the number of such queries is around 400-1000. So each query will result in writing many records that lie in its range.
So which of the following is faster approach?
1- As soon as I start getting my query range records, I start writing them in File serially?
2- I store my records in some string, and append in string for each record found, and then in the end write it?
3- Use string builder to concatenate result of each query and then write to file at the end? but string builder doesnot support new line.
4- Make my own datastrucuture ?
5- Make an array list of records and then iterate over it at the end to write to file?
Please let me know which approach is fastest both in terms of IO and MM.
OS level write buffering is quite effective. Go with #1 unless you have profiler data suggesting there’s a problem.