Question

I’m using a Java executor to divide and process a given task in parallel. There could be any number of tasks. They are all queued at the beginning and the master controller object simply waits for all of them to finish.

The problem I’m running into is how to determine when all tasks have been completed. Since one large task is only every queued at a single time (i.e. all currently queued tasks all belong to the same master task) I can use the getCompletedTaskCount() method to compare the number of completed tasks to the number of the number of tasks originally queued.

However, this requires that I constantly poll the executor for the number of completed tasks and in my opinion isn’t that great of a solution.

while (pool.getCompletedTaskCount() - start_count < num_tasks)
{
    try
    {
        Thread.sleep(30);
    }
    catch (InterruptedException e)
    {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }
}

I thought about having a counter object which each task could increment, then if the count is equivalent to the number of expected tasks notify the master thread. Something like this (ignore the somewhat incomplete code like missing exception handlers and such):

Master thread:

counter_object.expected_count = num_tasks;
counter_object.count = 0;
queue_tasks();
synchronized(counter_object)
{
    counter_object.wait();
}
// all tasks have finished

Worker tasks:

// ...do task
// task finished, update counter
synchronized(counter_object)
{
    ++counter_object.count;
    if(counter_object.count == counter_object.expected_count)
    {
        // all tasks have finished, notify master thread
        counter_object.notify();
    }
}

This method also has the added benefit that I can use a single executor to run multiple master tasks since the counter object would be local to a given master.

Is there a better way to solve this problem? The number of tasks might very well be larger than the max number of threads the executor is allowed to create so I don’t think a CyclicBarrier would work.

Answer 1

This sounds like a job of ExecutorService.invokeAll.

Collection<Callable> tasks = <get all sub tasks>;
executorService.invokeAll(tasks);
// Execution proceeds at the following line only once all "tasks" have been run

Alternatively (since you are likely dealing with Runnable instances and not Callable, you can use ExecutorService.submit(Runnable) and then wait for them to complete.

for (Runnable task:tasks) {
    futures.add(executorService.submit(task));
}

for (Future<Void> result:futures) {
    result.get();
}

Note: exception handling omitted