Question

I’m trying to find a better way to do this as it could take years to compute! I’m need to compute a map which is too large to fit in memory, so I am trying to make use of IO as follows.

I have a file that contains a list of Ints, about 1 million of them. I have another file that contains data about my (500,000) document collection. I need to calculate a function of the count, for every Int in the first file, of how many documents (lines in the second) it appears in. Let me give an example:

File1:

-1
1
2
etc...

file2:

E01JY3-615,  CR93E-177 , [-1 -> 2,1 -> 1,2 -> 2,3 -> 2,4 -> 2,8 -> 2,... // truncated for brevity] 
E01JY3-615,  CR93E-177 , [1 -> 2,2 -> 2,4 -> 2,5 -> 2,8 -> 2,... // truncated for brevity]
etc...

Here is what I have tried so far

def printToFile(f: java.io.File)(op: java.io.PrintWriter => Unit) {
    val p = new java.io.PrintWriter(new BufferedWriter((new FileWriter(f))))
    try {
      op(p)
    } finally {
      p.close()
    }
  }

  def binarySearch(array: Array[String], word: Int):Boolean = array match {
    case Array() => false
    case xs      => if (array(array.size/2).split("->")(0).trim().toInt == word) {
      return true
    } else if (array(array.size/2).split("->")(0).trim().toInt > word){
      return binarySearch(array.take(array.size/2), word)
    } else {
      return binarySearch(array.drop(array.size/2 + 1), word)
    }
  }

  var v = Source.fromFile("vocabulary.csv").getLines()

  printToFile(new File("idf.csv"))(out => {
    v.foreach(word =>{
      var docCount: Int = 0
      val s = Source.fromFile("documents.csv").getLines()
      s.foreach(line => {
        val split = line.split("\\[")
        val fpStr = split(1).init
        docCount = if (binarySearch(fpStr.split(","), word.trim().toInt)) docCount + 1 else docCount
      })
      val output = word + ", " + math.log10(500448 / (docCount + 1))
      out.println(output)
      println(output)
    })
  })

There must be a faster way to do this, can anyone think of a way?

Answer 1

From what I understand of your code, you are trying to find every word in the dictionary in the document list.
Hence, you are making N*M comparisons, where N is the number of words (in the dictionary with integers) and M is the number of documents in the document list. Instantiating to your values, you are trying to calculate 10^6 * 5*10^5 comparisons which is 5*10^11. Unfeasible.

Why not create a mutable map with all the integers in the dictionary as keys (1000000 ints in memory is roughly 3.8M from my measurements) and pass through the document list only once, where for each document you extract the integers and increment the respective count values in the map (for which the integer is key).

Something like this:

import collection.mutable.Map
import scala.util.Random._

val maxValue = 1000000

val documents = collection.mutable.Map[String,List[(Int,Int)]]()

// util function just to insert fake input; disregard
def provideRandom(key:String) ={ (1 to nextInt(4)).foreach(_ => documents.put(key,(nextInt(maxValue),nextInt(maxValue)) :: documents.getOrElse(key,Nil)))}

// inserting fake documents into our fake Document map 
(1 to 500000).foreach(_ => {val key = nextString(5); provideRandom(key)})

// word count map
val wCount = collection.mutable.Map[Int,Int]()

// Counting the numbers and incrementing them in the map
documents.foreach(doc => doc._2.foreach(k => wCount.put(k._1, (wCount.getOrElse(k._1,0)+1))))

scala> wCount
res5: scala.collection.mutable.Map[Int,Int] = Map(188858 -> 1, 178569 -> 2, 437576 -> 2, 660074 -> 2, 271888 -> 2, 721076 -> 1, 577416 -> 1, 77760 -> 2, 67471 -> 1, 804106 -> 2, 185283 -> 1, 41623 -> 1, 943946 -> 1, 778258 -> 2...

the result is a map with its keys being a number in the dict and the value the number of times it appears in the document list

This is oversimplified since

• I dont verify if the number exists in the dictionary, although you only need to init the map with the values and then increment the value in the final map if it has that key;
• I dont do IO, which speeds up the whole thing

This way you only pass through the documents once, which makes the task feasible again.