Question

I am using:

PrintWriter out = new PrintWriter(new BufferedWriter(new FileWriter("test.txt"),1024*1024*500))

to write a large file (approx. 2GB). It takes 26 seconds to write. But, when I replace 500 with 10/20, it takes 19 seconds.

From here, what I understood is buffering gives better performance. If so, then why is this happening? I checked it by running 5 times each, so system/IO load is not an issue.

Answer 1

As I said in a previous question, there is an optimal buffer size (which is typically around 32 KB) and as you make the buffer larger than this it is slower not faster. The default buffer size is 8 KB.

BTW: How large is your L2/L3 CPU cache? (about 10 MB I suspect) Your primary L1 cache is about 32 KB?

By using a buffer which fits into the fastest cache, you are using the fastest memory. By using a buffer which only fits in main memory, you are using the slowest memory (as much as 10x slower)

---

In answer to your question.

What I do is assume ISO-8859-1 encoding i.e (byte) ch and write a byte at a time to a ByteBuffer, possibly memory mapped.

I have methods for writing/reading long and double from a ByteBuffer without creating any garbage.

https://github.com/peter-lawrey/Java-Chronicle/blob/master/src/main/java/vanilla/java/chronicle/impl/AbstractExcerpt.java

Using this approach you can log about 5 million lines per second to disk.