Question

I have a collection in my MongoDB database that had Mongoid::Versioning enabled for it quite sometime ago. Unfortunately, it made some of my documents extremely large in size. I see some that are over 711K. This makes for expensive disk i/o and expensive read/write times. I am looking for a solution to go through this collection (which has almost 2 million documents), and remove all mongoid versioning, safely if possible. From what I can tell, Mongoid just stores the versions in an array attribute named just that, versions. If there is a way to gank it from all of my documents in a way that won’t completely make the database unusable (in terms of performance while I do an entire disk scan + write/update), that would be great.

Answer 1

There is a lot of ways to handle this situation. I’ve tried this a couple of different ways, and for an trial of ten thousand records they have similar processing times. I’ve tried another and found it much worse. I’ll attach them here in case it helps.

Here I am working on the hypothesis that batching the process up will help alleviate the impact on your database.

The first method would be to do finds on the collection, with a limit to handle a batch.

var batchsize = 50
var c = db.collection.count()
for(x=0;x<Math.floor(c/batchsize);x++){
    db.collection.find({versions: {$exists:true}}).limit(batchsize).forEach(function(cur){
        db.collection.update({_id:cur._id},{$unset:{versions:""}})
    })
}

The issue here will be the collection scans that will be required on every new batch. The limit will help with the impact, but it is still costly on the collection.

A second method would be to fill an array with the _ids of all the documents which have a versions array, then iterate through the array and update:

var arr = db.collection.find({versions:{$exists:true}},{_id:1}).toArray()
while(arr.length>0){
    for(x=0;x<batchsize;x++){
        var curId = arr.pop();
        db.collection.update(curId,{$unset:{versions:""}})
    }
}

This will mean an initial full collection scan, but after this point it is all iterating through the array and updating in batches.

I’ve tried a third method, where I work through the collection finding an _id greater than the previous one and updating, but found this to be much more expensive (even though it was able to use the index on _id). I’m adding it here in case it is useful.

var curid = db.collection.find({_id:{$gt:MinKey}},{_id:1}).sort({_id:1}).limit(1).next()._id;
while(curid < MaxKey){
    db.collection.update({_id:curid},{$unset:{versions:""}});
    curid = db.collection.find({_id:{$gt:curid}},{_id:1}).sort({_id:1}).limit(1).next()._id;
}