Question

A not uncommon pattern in Project Euler problems seems to be something equivalent to:

Stream.from(1).map(f).takeWhile((_>0)).foldLeft(0L)(_+_)

where f is some expensive-to-compute function which returns positive values up to some unknown point, and returns zeros after that.

I do like parallelising things, especially when Scala’s parallel collections and .par make it so easy. But in the absence of a ParStream, the best I’ve come up with is:

val BATCH=1024
Stream.from(1,BATCH).flatMap(
  i=>(i until i+BATCH).par.map(f)
).takeWhile((_>0)).foldLeft(0L)(_+_)

which doesn’t seem very elegant and is sensitive to the choice of BATCH value (but can yield x4 speed improvements on my quad-core).

Any suggestions for cleaner ways of achieving the same result ?

Answer 1

Well, you can make it a bit more elegant with something like Stream from 1 grouped BATCH map (_.par), though I’d rather use Iterator in this case — much lighter weight than Stream, and it won’t fill up REPL’s memory as I experiment with it.