Question

I am new to mongoDb and planning to use map reduce for computing large amount of data.
As you know we have map function to match the criteria and then emit the required data for a given filed. In my map function I have multiple emits. As of now I have 50 Fields emitted from a single document. That means from a single document in a collection explodes to 40 document in temp table. So if I have 1 million documents to be processed it will 1million * 40 documents in temp table by end of map function.

The next step is to sort on this collection. (I haven’t used sort param of map will it help?)

Thought of splitting the map function into two….but then one more problem …while performing map function if by chance I ran into an exception thought of skipping entire document data (I.e not to emit any data from that document) but if I split I won’t be able to….

In mongoDB.org i found a comment which said…”When I run MR job, with sort – it takes 1.5 days to reach 23% at first stage of MR. When I run MR job, without sort, it takes about 24-36 hours for all job.Also when turn off jsMode is speed up my MR twice ( before i turn off sorting )”

Will enabling sort help? or Will turning OFF jsmode help? i am using mongo 2.0.5

Any suggestion?

Thanks in advance .G

Answer 1

The next step is to sort on this collection. (I haven’t used sort param of map will it help?)

Don’t know what you mean, MR’s don’t have sort params, only the incoming query has a sort param. The sort param of the incoming query only sorts the data going in. Unless you are looking for some specific behaviour that will avoid sorting the final output using an incoming sort you don’t normally need to sort.

How are you looking to use this MR. Obviusly it won’t be in realtime else you would just kill your servers so Ima guess it is a background process that runs and formats data to the way you want. I would suggest looking into incremental MRs so that you do delta updates throughout the day to limit the amount of resources used at any given time.

So if I have 1 million documents to be processed it will 1million * 40 documents in temp table by end of map function.

Are you emiting multiple times? If not then the temp table should have only one key per row with documents of the format:

{
_id: emitted_id
[{ //each of your docs that you emit }]
}

This is shown: http://kylebanker.com/blog/2009/12/mongodb-map-reduce-basics/

or Will turning OFF jsmode help? i am using mongo 2.0.5

Turning off jsmode is unlikely to do anything significant and results from it have varied.