Question

I have been looking into scala and AKKA for managing an obviously parallelisable algorithm.
I have some knowledge of functional programming and mostly do Java, so my FP might not be the best yet.

The algorithm I am working with is pretty simple, there is a top computation:

  def computeFull(...): FullObject

This computation calls sub computations and then sum it up (to simplify):

  def computePartial(...): Int

and computeFull does something like this (again simplifying):

 val partials = for(x <- 1 to 10
     y <- 1 to 10) yield computePartial(x, y)
 partials.foldLeft(0)(_ + _)

So, it’s very close to the AKKA example, doing the PI computation. I have many computeFull to call and many computePartial within each of them. So I can wrap all of this in AKKA actors, or to simplify in Futures, calling each computeFull and each computePartial in separate threads. I then can use the fold, zip and map functions of http://doc.akka.io/docs/akka/snapshot/scala/futures.html to combile the futures.

However, this implies that computeFull and computePartial will have to return Futures wrapping the actual results. They thus become dependent on AKKA and assuming that things are run in parallel. In fact, I also have to implicitly pass down the execution contexts within my functions.

I think that this is weird and that the algorithm “shouldn’t” know the details of how it is parallelised, or if it is.

After reading about Futures in scala (and not the AKKA one) and looking into Code Continuation. It seems like the Responder monad that is provided by scala (http://www.scala-lang.org/api/current/scala/Responder.html) seems like the right way to abstract how the function calls are run.
I have this vague intuition that computeFull and computePartial could return Responders instead of futures and that when the monad in executed, it decides how the code embedded within the Responder gets executed (if it spawns a new actor or if it is executed on the same thread).

However, I am not really sure how to get to this result. Any suggestions? Do you think I am on the right way?

Answer 1

If you don’t want to be dependent on Akka (but note that Akka-style futures will be moved and included with Scala 2.10) and your computation is a simple fold on a collection you can simply use Scala’s parallel collections:

val partials = for { x <- (1 to 10).par
  y <- 1 to 10
} yield computePartial(x, y)
// blocks until everything is computed
partials.foldLeft(0)(_ + _)

Of course, this will block until partials is ready, so it may not be a appropriate situation when you really need futures.

With Scala 2.10 style futures you can make that completely asynchronous without your algorithms ever noticing it:

def computePartial(x: Int, y: Int) = {
  Thread.sleep((1000 * math.random).toInt)
  println (x + ", " + y)
  x * y
}

import scala.concurrent.future
import scala.concurrent.Future
val partials: IndexedSeq[Future[Int]] = for {
  x <- 1 to 10
  y <- 1 to 10
} yield future(computePartial(x, y))

val futureResult: Future[Int] = Future.sequence(partials).map(_.fold(0)(_ + _))

def useResult(result: Int) = println(s"The sum is $result")

// now I want to use the result of my computation
futureResult map { result => // called when ready
  useResult(result)
}
// still no blocking
println("This is probably printed before the sum is calculated.")

So, computePartial does not need to know anything about how it is being executed. (It should not have any side-effects though, even though for the purpose of this example, a println side-effect was included.)

A possible computeFull method should manage the algorithm and as such know about Futures or parallelism. After all this is part of the algorithm.

(As for the Responder: Scala’s old futures use it so I don’t know where this is going. – And isn’t an execution context exactly the means of configuration you are looking for?)