Question

I am working on a service that returns a List<Response> object.

There are two kind of object that can build the response. One of them populates the content from internal data, the other reaches out to a service and gets the data.

for(Item item : items){
   if(item is external){
     call service and get result and populate Response and add to List
   }

   if(item is internal)
   {
     populate response object and add to list
   }
}

Current impl is procedural and blocking, what i want is a non blocking design such that, if the item is external fire the call and continue with next item in the list. Then when the loop is done, i can wait for all to finish.

What would be a good approach to do this? I m also considering creating seperate classes per responsibility.

EDIT: the reason to introduce is to reduce latency hits.

Answer 1

ExecutorCompletionService (thread pool with callbacks that remembers submitted tasks) seems like a goof git:

CompletionService<Response> executorService = 
    new ExecutorCompletionService<>(Executors.newFixedThreadPool(10));
int totalExternal = 0;

for(Item item : items){
    if(item is external){
        executorService.submit(externalCall(item));
        ++totalExternal;
    }

    if(item is internal){
        populate response object and add to list
    }
}

for (int i = 0; i < totalExternal; ++totalExternal) {
    addAsynchResultToResponseList(executorService.take().get());
}

Where externalCall(item) is defined as follows for clarity:

Callable<Response> externalCall(Item item) {
  return new Callable<Response>() {
    //...
  }
}

Obviously once you go asynchronous the result list can have an arbitrary order.

---

Another approach would be to use ordinary ExecutorService and have an intermediate List<Future<Response>>. The trick is to use AsyncResult wrapper to wrap internal responses (it creates Future that is immediately done and returns passed value).

List<Future<Response>> futures = new ArrayList<>();
for(Item item : items){
    if(item is external){
        futures.add(executorService.submit(externalCall(item)));
    }

    if(item is internal){
        futures.add(new AsyncResult(synchResponse));
    }
}

Now you can simply iterate over futures. AsyncResult will return immediately as the value was already computer when it was created (synchResponse). But you will have to wait for Futures returned from a thread pool.

Remember that Future.get() allows you to retrieve original exception. Also the order of Futures in the list is the same as the order of original items, so if nth Future fails, nth item in items list was the cause.