Question

The problem is:

I have 3 mongo tables, and I want to migrate data from it to 3 SQL Server tables. Each mongo table has only one index on _id.

I use official C# driver to do this. I use a cursor to read each mongo tables,just like:

MongoCursor mongocursor = mongoCollection.find();
mongocursor.setHint("_id_");
mongocursor.setSortOrder(new string[] { "_id" });
foreach (BsonDocument bd in mongocursor)
{
     //To do sth...
}

Then I use mongostat to monitor the query. I notice each getmore command has become slower and slower. After about 10mins with migrating 15,000,000 records, each getmore command will take about 5s. Then after about 20mins with migrating 30,000,000 records, each getmore command will take about 10s!

Log is like this:

11:04:19 [conn809] getmore
YWANG4TestDataBase.UserOperationLog_2012_03_17 query: { $query: {},
$hint: "`_id_`", $orderby: { _id: 1 } } cursorid:80356003137218
nreturned:10396 reslen:4194659 8578ms

Please help me. I don’t know what happened.

Answer 1

If you’re querying for all documents in the collection, you will actually get better performance by skipping the index and the sort (unless the sort order is important for your import process). By skipping the sort, you will retrieve results in on-disk order (which is undefined for normal collections), and may be able to avoid a lot of disk seeking on the MongoDB server.