Question

I have a groovy script which uses the Mongo Java Driver mongo-java-driver-2.8.0.jar to visit all there records in a single collection updating any that do not match the expected structure. The script works like a champ, but I am left scratching my head why it processes more records than the collection actually has. Or, more accurately, the dbCursore.hasNext() iterates over more records than the collection actually has. This only occurs if the script finds something to update. If the script executes with no updates, the total number processed is correct.

Does hasNext() “start over” or do records move within the iteration if they’ve been updated?

Here’s the code…

static def doIt( mongo, normalizer, isDryRun ) {
    def ttlProcessed = 0
    def ttlCandidates = 0
    def ttlUpdated = 0
    def lapCount = 0;
    def lapStartTime = System.currentTimeMillis();

    def db = mongo.getDB( "devices" )
    DBCollection dbCollection = db.getCollection( "profiles" )
    DBCursor dbCursor = dbCollection.find();
    while ( dbCursor.hasNext() ) {
        DBObject source = dbCursor.next();
        DBObject normalized = normalizer.normalize( source )
        // Only update if changed...
        if ( ! ( source.equals( normalized ) ) ) {
            ttlCandidates++
            if ( !isDryRun ) {
                BasicDBObject searchQuery = new BasicDBObject( "_id", normalized.get( "_id" ) )
                WriteResult result = dbCollection.update( searchQuery, normalized, false, false, WriteConcern.SAFE );
                ttlUpdated++
            }
        }
        ttlProcessed++;
        if ( ttlProcessed % 10000 == 0 ) {
            printErr "split: ${lapCount}, splitElapsed: ${calcElapsed( lapStartTime) } ms, elapsed: ${calcElapsed( startTime )} ms, processed: ${ttlProcessed}, candidates: ${ttlCandidates}, updated: ${ttlUpdated}"
            lapCount++
            lapStartTime = System.currentTimeMillis()
        }
    }
    printErr "split: ${lapCount}, splitElapsed: ${calcElapsed( lapStartTime) } ms, elapsed: ${calcElapsed( startTime )} ms, processed: ${ttlProcessed}, candidates: ${ttlCandidates}, updated: ${ttlUpdated}"
}

How is it that ttlProcessed gets a value higher than the count of the collection being processed if the run updates any records?

Answer 1

This can happen because an update causes a document to move (usually because of growth). If a document does grow and is processed again, it will count as an iteration, but assuming your update is idempotent (I have not tested), then it will not be an issue.

You can use the $snapshot option to get around this if it is a concern. I would also recommend reading this:

http://www.mongodb.org/display/DOCS/How+to+do+Snapshotted+Queries+in+the+Mongo+Database

These issues are essentially why tools like mongodump and mongoexport walk the _id index (i.e. use $snapshot by default).

If you want to look into preventing the moves in the first place, then take a look at this section on the Padding Factor page:

http://www.mongodb.org/display/DOCS/Padding+Factor#PaddingFactor-ManualPadding

There are also options in 2.2 to set padding when running the compact command:

http://docs.mongodb.org/manual/release-notes/2.2/#padding-specifiable-on-compact-command