Question

I’m trying to modify a serial program that reads from a database, and writes results to a file, this is done in a blocking way and I think we can get performance boost of there is a memory buffer and have the file being written in the “background” asynchronously

I can think of the “job interview” solution, using Threads, shared resource, syncronized blocks etc…
but I’m sure there is a better way (is there a nice little “delayed write” library out there that will do it for me?)

Does any of the java.util.concurrent package offer any help? java.nio? or perhaps JMS/ActiveMQ?

What about PipedOutputStream / PipedInputStream as a basis for my buffer?

How do I implement a delayed / background / buffered / non-blocking / asynchronous file writter in Java?

Edit:

Based on the suggestions, and to avoid having this question closed, (as I think it’s still relevant based on the answers comments and votes) here is an attempt to make it more focused. (I’m keeping the original question above so the answers will still remain in context, as there are some very good ones)

• What are the practical differences between PipedOutputStream / PipedInputStream (or PipedReader / PipedWriter) and BlockingQueue
• and is it preferable to use the latter for asynchronous file writing?
  (Or is this apples to oranges comparison and if so, I’d like to know why?)

Answer 1

You probably want to use a bounded blocking queue between the producer (the database reader) and the consumer (the file writer).

Java’s ArrayBlockingQueue does the job quite nicely. The producer blocks if the buffer is full, avoiding any issues with consuming too much memory.

Doing the producing and consuming in concurrent threads is best achieved using Java’s Executors framework.