Question

Yesterday, this piece of code caused me a headache. I fixed it by reading the file line by line. Any ideas ?

The while loop never seems to get executed even though the no of lines in the file is greater than 1.

 val lines = Source.fromFile( new File("file.txt") ).getLines;

 println( "total lines:"+lines.size );

 var starti = 1;
 while( starti < lines.size ){
   val nexti = Math.min( starti + 10, lines.size  );

   println( "batch ("+starti+", "+nexti+") total:" + lines.size )
   val linesSub = lines.slice(starti, nexti)
   //do something with linesSub
   starti = nexti
 }

Answer 1

This is indeed tricky, and I would even say it’s a bug in Iterator. getLines returns an Iterator which proceeds lazily. So what seems to happen is that if you ask for lines.size the iterator goes through the whole file to count the lines. Afterwards, it’s “exhausted”:

scala> val lines = io.Source.fromFile(new java.io.File("....txt")).getLines
lines: Iterator[String] = non-empty iterator

scala> lines.size
res4: Int = 15

scala> lines.size
res5: Int = 0

scala> lines.hasNext
res6: Boolean = false

You see, when you execute size twice, the result is zero.

There are two solutions, either you force the iterator into something ‘stable’, like lines.toSeq. Or you forget about size and do the “normal” iteration:

while(lines.hasNext) {
  val linesSub = lines.take(10)
  println("batch:" + linesSub.size)
  // do something with linesSub
}