Question

I am developing an application in which I continuously receive messages. I am storing these messages in an in-memory data structure (say List). I want to write those messages to a file but only after list size reaches some threshold value say 100 messages (perform batch processing on the messages).
One way is I can simply check the list size after every message is received and call a function to write messages to a file if Threshold value is reached. But the problem with this approach is:

1. Calling function may need to wait indefinitely until all the messages are written to the file
2. The incoming messages may be lost in the process or might need to wait for getting stored in the List.

Other way could be to spawn a new thread, which will write messages to a file independently. But when I pass the list (containing messages) to the thread for performing write operation, it gets updated with new messages which are continuously coming in. As a result newly arrived messages also gets written to the file which is not expected.

This should not happen as I am intending the new messages to be written in the next batch.

Can someone suggest me a solution for this requirement, or any improvements in the above approach that can solve my issues.

Answer 1

It’s important to understand that you never pass an object around in Java – only ever a reference (or a primitive value).

Options:

• Create a copy of the list, and pass a reference to that copy to your new thread
• Use a producer/consumer queue, so your “producing” thread only ever adds values to the queue, and your consumer thread only ever takes items from the queue to write them to disk. You’ll want to think about how large you want the queue to potentially get before it stops accepting more entries, of course.

I’d recommend the latter approach, using the classes in the java.util.concurrent package to implement it; particularly BlockingQueue<E> implementations.