Question

I insert bunch of docs into one collection, then a cron job moves them to another collection.

The reason I do this is that there is a non-indexed collection that needs to store data fast, the second collection is indexed and hence slow for inserts, so the cron job moves the data.

I use the following script to move data

db.smalldaily.find().forEach(function(doc){db.largedaily.save(doc);db.smalldaily.remove(doc);});

The problem is right after moving the data is completed, mongod seems to crash, when I log into the mongo shell and type db.large and then press the TAB button for auto-completing the collection name, mongodb goes to coma and I need to restart the mongod service to make it work.

Am I doing something wrong or the rumors that mongodb is still immature are partially true?

I’m using MongoDB shell version: 2.2.1 on CentOS

Answer 1

So, this is basically running as server side javascript that creates a massive forEach loop in the shell iterating across those 1 million+ documents and I am betting that the amount of (non-mapped) memory is sky high while this is running.

If you ran this outside the shell using a driver instead you would likely not see the problems at all. Does the migration complete and then not clean up the memory? If so, then that is probably the issue.

Changes are coming in the next release (2.3+) to make the shell far more performant, but for now it would probably be a better idea to do this via a driver instead of internally in the shell. For one thing it would immediately be more efficient to do this in batches rather than iterating one at a time.

With all that being said, if this is easily reproducible, and especially if you can provide an example data set, this would be perfect fodder for a bug report so that the developers can figure out why the shell is barfing when processing this loop.