Question

I know about the parallel collections in Scala. They are handy! However, I would like to iterate over the lines of a file that is too large for memory in parallel. I could create threads and set up a lock over a Scanner, for example, but it would be great if I could run code such as:

Source.fromFile(path).getLines.par foreach { line =>

Unfortunately, however

error: value par is not a member of Iterator[String]

What is the easiest way to accomplish some parallelism here? For now, I will read in somes lines and handle them in parallel.

Answer 1

You could use grouping to easily slice the iterator into chunks you can load into memory and then process in parallel.

val chunkSize = 128 * 1024
val iterator = Source.fromFile(path).getLines.grouped(chunkSize)
iterator.foreach { lines => 
    lines.par.foreach { line => process(line) }
}

In my opinion, something like this is the simplest way to do it.