Question

I want to use IO monad.

But this code do not run with large file.
I am getting a StackOverflowError.
I tried the -DXss option, but it throws the same error.

val main = for {
  l <- getFileLines(file)(collect[String, List]).map(_.run)
  _ <- l.traverse_(putStrLn)
} yield ()

How can I do it?

---

I wrote Iteratee that is output all element.

def putStrLn[E: Show]: IterV[E, IO[Unit]] = {
  import IterV._
  def step(i: IO[Unit])(input: Input[E]): IterV[E, IO[Unit]] =
    input(el = e => Cont(step(i >|> effects.putStrLn(e.shows))),
      empty = Cont(step(i)),
          eof = Done(i, EOF[E]))
  Cont(step(mzero[IO[Unit]]))
}
val main = for {
  i <- getFileLines(file)(putStrLn).map(_.run)
} yield i.unsafePerformIO

This is also the same result.

I think to be caused by IO implementation.

Answer 1

This is because scalac is not optimizing loop inside getReaderLines for tail calls. loop is tail recursive but I think the case anonymous function syntax gets in the way.

Edit: actually it’s not even tail recursive (the wrapping in the IO monad) causes at least one more call after the recursive call. When I was doing my testing yesterday, I was using similar code but I had dropped the IO monad and it was then possible to make the Iteratee tail recursive. The text below, assumes no IO monad…

I happened to find that out yesterday while experimenting with iteratees. I think changing the signature of loop to this will help (so for the time being you may have to reimplement getFilesLines and getReaderLines:

@annotations.tailrec
def loop(it: IterV[String, A]): IO[IterV[String, A]] = it match {
  // ...
}

We should probably report this to the scalaz folk (and may be open an enhancement ticket for scala).

This shows what happens (code vaguely similar to getReaderLines.loop):

@annotation.tailrec
def f(i: Int): Int = i match {
  case 0 => 0
  case x => f(x - 1)
}
// f: (i: Int)Int

@annotation.tailrec
def g: Int => Int = {
  case 0 => 0
  case x => g(x - 1)
}
/* error: could not optimize @tailrec annotated method g: 
it contains a recursive call not in tail position
       def g: Int => Int = {
                           ^
*/