Question

I’m looking to compare fields between potentially millions of documents within a mongo collection. The fields will be determined ahead of time and weights will be given to each field. These weights will then be used to return document pairs representing suggestions for ‘like’ documents. For instance, if 2 documents are being compared and both have the same value for the field ‘first_name’, the weight table will be referenced and the score for the pair will have that weight added to it. If another field is the same between the two, the score will updated to reflect a higher likeness.

I’m currently approaching this by iterating through the initial result set, then having an embedded iteration that also goes through the result set and compares each document to the document that the first iterator is on (extremely inefficient). This is currently all done by php as it grabs elements through the cursor.

I’m open for any suggestions including MapReduce implementations (doesn’t really seem applicable), cursor manipulation, pretty much whatever you can conjure up to simplify the process because im working at O(n^2) complexity right now (Well, a little better as I skip the documents that have been covered so far by the first iterator).

Answer 1

To avoid n^2 you would have to look at storing fields and their values in a reference collection, e.g. :

{
    field: "firstName",
    value: "Remon",
    documents : [ <list with all document _ids of documents that have "field" set to "value">]
}

This way you can query directly on this collection to get all documents that are “like” your source document. Additionally this allows you to query for multiple key/value pairs with a single O(n) query.

Obviously the only tricky thing is maintaining this reference collection in the first place but in your case that seems pretty straightforward (update references when you update the fields).

Does that help?