Question

I have a map reduce like this:

map:

function() {
        emit(this.username, {sent:this.sent, received:this.received});
}

reduce:

function(key, values) {                                                                                                                                                                                                         
    var result = {sent: 0, received: 0, entries:0};

    values.forEach(function (value) {
        result.sent += value.sent;
        result.received += value.received;
        result.entries += 1;
    });

    return result;
}

I’ve been monitoring the amount of entries processed in the result map, as you can see. I’ve found I get much lower numbers of accessed records than I should.

For my particular data set, the output is like so:

[{u’_id’: u’1743′, u’value’: {u’received’: 1406545.0, u’sent’: 26251138.0, u’entries’: 316.0}}]

As I’m running the map reduce with a query option, specifying a username and a date range.

If I perform the same query using db.collection.find() as follows, the count is different:

> db.entire_database.find({username: '1743', time : { $lte: ISODate('2011-08-12 12:40:00'), $gte: ISODate('2011-08-12 08:40:00') }}).count()                        
1915

The full map reduce query is this:

db.entire_database.mapReduce(m, r, {out: 'myoutput', query: { username: '1743', time : { $lte: ISODate('2011-08-12 12:40:00'), $gte: ISODate('2011-08-12 08:40:00') } }  })

So basically, I’m unsure why the count is so radically different? Why is the find() giving me 1915, but the map reduce is 316?

Answer 1

Your map function needs to emit an object with the same form as the reduce function (ie. it should have an entries field set to 1). You can read more about this here.

Basically, the values that are passed to the reduce function are not necessarily the raw outputs emitted from map. Rather than being called once, the reduce function is called many times on ‘groups’ of values produced by map, the results of which are then combined again by being passed into a further call of the reduce function. This is what makes MapReduce horizontally scalable, because any group of emitted values can be farmed out to any server in any order before being combined later.

So I would restructure your functions slightly like this:

map:

function() {
    emit(this.username, {sent:this.sent, received:this.received, entries : 1});
}

reduce:

function(key, values) {                                                                                                                                                                                                         
    var result = {sent: 0, received: 0, entries:0};

    values.forEach(function (value) {
        result.sent += value.sent;
        result.received += value.received;
        result.entries += value.entries;
    });

    return result;
}