Question

I have a problem with a python-script witch i use to aggregate large collections into smaller pieces and have them grouped by a timestamp.

map = Code("function(number) {"
    "emit({"
        "ts : new Date(new Date((this.ts - (this.ts % (60 * number))) * 1000).setSeconds(0))"
   "}, 1);"
"}")

reduce = Code("function(key, vals) {"
    "var sum = 0;"
    "for (var i in vals) {"
        "sum += vals[i]"
    "}"
    "return sum;"
"}")

As you can see, it’s a pretty simple MapReduce and the timestamp (ts) should be grouped by anny given number of minutes. I’ve tested the Javascript here http://jsfiddle.net/QgMzK/1/ and it seems to work fine. But when i run it in Python all timestamps become ISODate(“1970-01-01T00:00:00Z”).

Any ideas?

Answer 1

Your map function takes one parameter: number, which, when called by map-reduce, will be set to null and after coertion (and some % by zero) will make the date your map is returning ISODate("0NaN-NaN-NaNTNaN:NaN:NaNZ"). This after type conversion becomes datetime.datetime(1970, 1, 1, 0, 0).

Remove the parameter and it should work.

edit

To confirm that, try running this code:

from pymongo import Connection
from bson.code import Code

db = Connection().mr_test
for i in xrange(10):
    db.things.insert({"x" : i})


map = Code("function(number) {"
    "emit({"
        "ts : number"
    "}, 1);"
"}")

reduce = Code("function(key, vals) {"
    "var sum = 0;"
    "for (var i in vals) {"
        "sum += vals[i]"
    "}"
    "return sum;"
"}")


result = db.things.map_reduce(map, reduce, "test_results")
for doc in result.find():
    print doc

The result on my machine is :

{u'_id': {u'ts': None}, u'value': 10.0}

Note that ts is None in the result, cause number was not set when the mapping function executed.

edit 2

AFAIK the only way to pass a parameter to map is by using scope optional parameter in map_reduce, but then you’ll have to remove it from map signature anyway.

So by changing your map to:

map = Code("function() {"
    "emit({"
        "ts : new Date(new Date((this.ts - (this.ts % (60 * number))) * 1000).setSeconds(0))"
    "}, 1);"
"}")

and by calling:

db.whatever.map_reduce(map, reduce, "collection_name", scope = {"number" : the_value_your_function_needs}) 

you can get the result you want.