Question

I’m trying to loop through all my documents in a particular collection and update the document (based on information already in the document). I started out doing this in python with pymongo, but it seems to be taking forever, so i decided to move it to the db box and run it in javascript, but it still seems to stall. I’m only talking about a coup,e thousand documents. What i seem to notice is:

start the task, first 500 objects get updated right away, few moments later, next 200, a little bit longer another 100, and then it just starts to slow down further and further.

If i remove the update call from my code, it flies through the loop just fine, so obviously it seems like my update is freezing something.

Heres the js i’m running:

mydb = db.getSisterDB(DB_NAME);

var cursor = mydb.user.find({is_registered:true}).limit(3000);
while (cursor.hasNext()) {
    var u = cursor.next();
    /* fix property on user doc */
    mydb.user.update({ _id: u._id }, u);
    print(cursor.objsLeftInBatch());
}

Answer 1

My guess is that the first 500 objects fit into some cache, so they are updated in memory and written to disk later. But after 500 write operation the cache is full, so subsequent writes have to wait until the previous changes have been written to the disk.

Your code reads the whole documents from the database, makes some change on them, and then sends the whole documents back to the database. This means that the whole document needs to be overwritten. It’s no miracle that this is slow.

You should use the $set operator in your update command to update only those fields which actually changed.

It would be even better performance-wise when you would manage to update the documents on the database without reading them into your application. You didn’t write what exactly you do with the documents and under what condition. When it isn’t that complex, this might be possible.