Question

I have a scientific application which I usually run in parallel with xargs, but this scheme incurs repeated JVM start costs and neglects cached file I/O and the JIT compiler. I’ve already adapted the code to use a thread pool, but I’m stuck on how to save my output.

The program (i.e. one thread of the new program) reads two files, does some processing and then prints the result to standard output. Currently, I’ve dealt with output by having each thread add its result string to a BlockingQueue. Another thread takes from the queue and writes to a file, as long as a Boolean flag is true. Then I awaitTermination and set the flag to false, triggering the file to close and the program to exit.

My solution seems a little kludgey; what is the simplest and best way to accomplish this?
How should I write primary result data from many threads to a single file?

The answer doesn’t need to be Java-specific if it is, for example, a broadly applicable method.

Update

I’m using “STOP” as the poison pill.

while (true) {
    String line = queue.take();
    if (line.equals("STOP")) {
        break;
    } else {
        output.write(line);
    }
}
output.close();

I manually start the queue-consuming thread, then add the jobs to the thread pool, wait for the jobs to finish and finally poison the queue and join the consumer thread.

Answer 1

That’s really the way you want to do it, have the threads put their output to the queue and then have the writer exhaust it.

The only thing you might want to do to make things a little cleaner is rather than checking a flag, simply put an “all done” token on to the queue that the writer can use to know that it’s finished. That way there’s no out of band signaling necessary.

That’s trivial to do, you can use an well known string, an enum, or simply a shared object.