Question

I have a very long sequence of strings which individually need to be processed by some processing function and then collected as another sequence object. The problem seems to be ideally suited to a fork/join type attack.

The function is a member of a class that is quite expensive to instantiate. But instantiating and sharing a single class object among the futures seemed to cause problems, so I instantiate 4 times the number of processors available and then split them up among the futures.

// instantiate processing class objects
val processors = 1 to SomeNumber map (x=> new MyProcessor)
val processorstream = Stream.continually(processors).flatten
// the input strings
val input: Seq[String] = some sequence of strings
val splitinput = input.grouped(some large number).toStream
// create futures
val mytask = splitinput.zip(processorstream).collect {
    case (subseq of strings, processor) => future {
        map elements of subsequence of strings with processor}}

Then I collect the output like so

val result = mytask.map(x => x.apply()).reduce(_++_) // or some appropriate concatenation operator

My problem is that this does not give me full cpu utilization even though I have 8 cores. It only utilizes one core.

To investigate, an alternative I tried was

val input: Seq[String] = some sequence of strings
// no stage where I split the input into subsequences
val mytask = input.zip(processorstream).collect {
    case (string, processor) => future {
        process string with processor}}
val result = mytask.map(x => x.apply())

This alternative both worked and didn’t work. It achieved full cpu utilization, but several exceptions were thrown because (a hypothesis) the processor was running through each string too fast and sometimes the same processor object would be applied to different strings simultaneously.

I’m even more sure of my hypothesis that the processors are working too fast because if I provide longer input (say, whole text documents instead of 10 word headlines) I get full cpu utilization without any exceptions thrown.

I’ve also tried experimenting with akka futures and scalaz promises and they all seem to only use one cpu when I split the input sequence into subsequences.

So how do I get full cpu utilization with futures in this instance while using subsequences of strings as input?

Answer 1

Per @om-nom-nom :

input.par.map { s => task(s) }