Question

I have this code snippet:

val chunks = blob.map(_.id).distinct.grouped(100).toSeq 
val output = (for (chunk <- chunks) yield { 
  MyDataStore.batchGet("tablez", chunk.asJava, classOf[TableZ], true).asScala }).flatten

I would like to convert this into a general purpose method which takes as parameters blob.map(_.id).distinct and MyDataStore.batchGet("tablez", chunk.asJava, classOf[TableZ], true).asScala

I am thinking maybe curried function but unsure about syntax.

Just to clarify, chunks is a Seq[Seq[String]], batchGet returns a List[T] where T is tablez, output is Seq[TableZ].

Answer 1

I think the following will work:

def getInGroups[T,U](ids: Seq[T], groupSize: Int = 100)(getter: Seq[T] => Seq[U]): Seq[U] = {
   ids.grouped(groupSize).flatMap{ chunk =>  getter(chunk) }.toSeq
}

Call it as

getInGroups(blob.map(_.id).distinct){ chunk => 
   MyDataStore.batchGet("tablez", chunk.asJava, classOf[TableZ], true).asScala
}