Question

I have some Mongoose Models with geospacial indexes:

var User = new Schema({
  "name" : String,
  "location" : {
     "id" : String,
     "name" : String,
     "loc" : { type : Array, index : '2d'}
  }
});

I’m trying to update all items that are in an area – for instance:

User.update({ "location.loc" : { "$near" : [ -122.4192, 37.7793 ], "$maxDistance" : 0.4 } }, { "foo" : "bar" },{ "multi" : true }, function(err){
    console.log("done!");
});

However, this appears to only update the first 100 records. Looking at the docs, it appears there is a native limit on finds on geospatial indices for that applies when you don’t set a limit.

(from docs:
Use limit() to specify a maximum number of points to return (a default limit of 100 applies if unspecified))

This appears to also apply to updates, regardless of the multi flag, which is a giant drag. If I apply an update, it only updates the first 100.

Right now the only way I can think of to get around this is to do something hideous like this:

Model.find({"location.loc" : { "$near" : [ -122.4192, 37.7793 ], "$maxDistance" : 0.4 } },{limit:0},function(err,results){
   var ids = results.map(function(r){ return r._id; });
   Model.update({"_id" : { $in : ids }},{"foo":"bar"},{multi:true},function(){
      console.log("I have enjoyed crippling your server.");
   });
});

While I’m not even entirely sure that would work (and it could be mildly optimized by only selecting the _id), I’d really like to avoid keeping an array of n ids in memory, as that number could get very large.

Edit:
The above hack doesn’t even work, looks like a find with {limit:0} still returns 100 results. So, in an act of sheer desperation and frustration, I have written a recursive method to paginate through ids, then return them so I can update using the above method. I have added the method as an answer below, but not accepted it in hopes that someone will find a better way.

This is a problem in mongo server core as far as I can tell, so mongoose and node-mongodb-native are not to blame. However, this is really stupid, as geospacial indices is one of the few reasons to use mongo over some other more robust NoSQL stores.

Is there a way to achieve this? Even in node-mongodb-native, or the mongo shell, I can’t seem to find a way to set (or in this case, remove by setting to 0) a limit on an update.

Answer 1

I’d love to see this issue fixed, but I can’t figure out a way to set a limit on an update, and after extensive research, it doesn’t appear to be possible. In addition, the hack in the question doesn’t even work, I still only get 100 records with a find and limit set to 0.

Until this is fixed in mongo, here’s how I’m getting around it: (!!WARNING: UGLY HACKS AHEAD:!!)

var getIdsPaginated = function(query,batch,callback){
  // set a default batch if it isn't passed.
  if(!callback){
    callback = batch;
    batch = 10000;
  }
  // define our array and a find method we can call recursively.
  var all = [],
      find = function(skip){
        // skip defaults to 0
        skip = skip || 0;
        this.find(query,['_id'],{limit:batch,skip:skip},function(err,items){
          if(err){
            // if an error is thrown, call back with it and how far we got in the array.
            callback(err,all);
          } else if(items && items.length){
            // if we returned any items, grab their ids and put them in the 'all' array
            var ids = items.map(function(i){ return i._id.toString(); });
            all = all.concat(ids);
            // recurse
            find.call(this,skip+batch);
          } else {
            // we have recursed and not returned any ids. This means we have them all.
            callback(err,all);
          }
        }.bind(this));
      };
  // start the recursion
  find.call(this);
}

This method will return a giant array of _ids. Because they are already indexed, it’s actually pretty fast, but it’s still calling the db many more times than is necessary. When this method calls back, you can do an update with the ids, like this:

Model.update(ids,{'foo':'bar'},{multi:true},function(err){ console.log('hooray, more than 100 records updated.'); });

This isn’t the most elegant way to solve this problem, you can tune it’s efficiency by setting the batch based on expected results, but obviously the ability to simply call update (or find for that matter) on $near queries without a limit would really help.