Question

How would I go about writing a co-occurence class in something like Java that takes a file full of n-grams and calculates word co-occurence for a given input term.

Are there any librarys or packages which work with Lucene (indexes) or something like a map-reduce over the n-gram list in Hadoop..?

Thanks.

Answer 1

Ok, so assuming you want to find the co-occurrence of two different words in a file of ngrams….

Here’s pseudo code-ish Java:

// Co-occurrence matrix
Hashmap<String,HashMap<String,Integer>> map = new HashMap();

// List of ngrams
ArrayList<ArrayList<String>> ngrams = ..... // assume we've loaded them into here already

// build the matrix
for(ArrayList<String> ngram:ngrams){
  // Calculate word co-occurrence in ngram for all words
  // result is an map strings-> count
  // words in alphabetical order
  Hashmap<String,<ArrayList<String>,Integer> wordCoocurrence = cooccurrence(ngram) // assume we have this

  // then just join this with original
}

// and just query with words in alphabetic order

Doing a count like this would probably be pretty with Pig but you’re probably more familiar with that than me