Question

As a C# programmer I have a sketchy understanding of Java / Scala iterator design.

I am trying to (lazily – for the source may be big) read records from a RecordReader (in some third party library). I need to do some additional work every 100 records.

for (group <- reader.iterator.zipWithIndex.grouped(100)) {
  for ((record, i) <- group) {
    println(i + "|" + record.key)
  }
  // ...
}

This gives me the very last record, repeatedly, each time.

If I don’t use grouped, it works fine and I get each record. Am I missing something about lazy streaming or Java iterators?

Answer 1

To troubleshoot, try to decorate your iterator in an another iterator that prints what is going on:

def wrap[T](i: Iterator[T]) = new Iterator[T] {
  def hasNext = { val b = i.hasNext; println("hasNext => " + b); b }
  def next() = { val n = i.next(); println("next() => " + n); n }
}

val reader = Iterator.from(20).take(10).toList
for (group <- wrap(reader.iterator).zipWithIndex.grouped(5)) {
  for ((v, i) <- group) println("[" + i + "] = " + v)
}

Call wrap on the iterator, the very first time you instantiate the iterator. This will print something like:

hasNext => true
hasNext => true
next() => 20
hasNext => true
next() => 21
hasNext => true

This should help you determine if the iterator is ill behaved… It could be for instance that the library does not deal correctly with calling hasNext multiple times without calling next. In that case you can modify wrap so that you make the iterator behave correctly. One more thing, from the symptoms, it feels like you’ve already consume the iterator before the grouped is called. So be extra careful and check if you’ve used the same iterator reference before.