Question

I have a PHP app that interacts with MongoDB. Until recently, the app was working fine but a few days back I found that the app is starting to respond REALLY slow. One of the collections has shot up to 500K+ records. So the MongCursor for any query on that collection keeps timing out.

I don’t think 500K records is WAY too much. Other pages using mongodb are beginning to slow down as well, but not as much as the one which uses the collection with 500k records. Static pages which don’t interact with MongoDB are still fast to respond.

I am not sure what could be the issue here. I have indexed the collections, so that does not seem to be a problem. Another point to note is that the RAM spec on the server is 512 MB and when PHP executes Mongo, top command show 15000k memory free.

Any help will be greatly appreciated.

Answer 1

To summarize followup from the chat room, the issue is actually related to a find() query which is doing a scan of all ~500k documents to find 15:

db.tweet_data.find({ 
    $or: 
    [ 
        { in_reply_to_screen_name: /^kunalnayyar$/i, handle: /^kaleycuoco$/i, id: { $gt: 0 } }, 
        { in_reply_to_screen_name: /^kaleycuoco$/i, handle: /^kunalnayyar$/i, id: { $gt: 0 } } 
    ], 
    in_reply_to_status_id_str: { $ne: null }
} ).explain() 
{ 
    "cursor" : "BtreeCursor id_1", 
    "nscanned" : 523248, 
    "nscannedObjects" : 523248, 
    "n" : 15, 
    "millis" : 23682, 
    "nYields" : 0, 
    "nChunkSkips" : 0, 
    "isMultiKey" : false, 
    "indexOnly" : false, 
    "indexBounds" : { 
        "id" : [ 
            [ 
                0, 
                1.7976931348623157e+308 
            ] 
        ] 
    } 
}

This query is using case-insensitive regular expressions which won’t make efficient use of an index (though there wasn’t actually one defined, in this case).

Suggested approach:

• create lowercase handle_lc and inreply_lc fields for search purposes

• add a compound index on those:

  db.tweet.ensureIndex({handle_lc:1, inreply_lc:1})

• the order of the compound index allows efficient find of all tweets either by handle or by (handle,in_reply_to)

• search by exact match instead of regex:


db.tweet_data.find({
$or:
[
{ in_reply_to_screen_name:'kunalnayyar', handle:'kaleycuoco', id: { $gt: 0 } },
{ in_reply_to_screen_name:'kaleycuoco', handle:'kunalnayyar', id: { $gt: 0 } }
],
})