Question

I have a program which writes some 8 millions line of data in a flat file. As of now, the program is calling bufferedwriter.write for each of the record and I was planning to write in bulk with the following strategy

1. Keep a data structure (I used array) to hold a specific number of records.

2. write the details in a file using the array. here is the code snippet (array is the name of the Array which stores the record and threshold count is the kick off for writing process)

   if (array.length==thresholdCount) {
       writeBulk(array);
   }
   
   public void writeBulk(String[] inpArray) {
        for (String line:inpArray) {
             if (line!=null) {
                  try {
                       writer.write(line +"\n"); 
                  } catch (IOException e) {
                       e.printStackTrace();
                  }
             }
        }
   
    }
   

However I am not seeing much performance improvement. I want to know if there is a way to determine the optimal threshold count?
I was also planning to further tune the code so as to store each element in the array as a concatenation of some n number of records and then call the bulk method. For ex an array with length 5000 will actually contain 50000 records whereby each index in the array contains 10 records. however before doing so, I need the expert opinion.

Answer 1

Writes to files are already buffered in a similar fashion before they are pushed to disk (unless you flush — which actually doesn’t always do exactly that either). Thus pre-buffering the writes will not speed up the overall process. Note: that some IO Classes try to do immediate writes by inserting flush requests after each write. For those special cases pre-buffering can sometimes help, but usually you just use a Buffered version of the Class in the first place rather than manually buffer yourself.

If you were writing to somewhere other than the end of the file, then you could see an improvement as writing to the middle of a file wouldn’t need to copy the contents of the already flushed entries sitting on your hard-disk.