Question

I’ve just started working with mongodb got a perl script that parses a twitter stream and adds it into mongodb. This part is all working fine.

However I’m stuck querying the database in PHP. I’ve got a combination of ajax and php to query the database and currently gets the last 30 entries. Now I’m wanting to have the page after so many seconds get the latest entries. With mysql database i would just pass the id back as a parameter and get the latest from database however I’ not quite sure how to approach it for mongodb.

The two options I’ve come up with are to use the number of documents in the collection and using that to get the latest documents skipping that many or to go back to the perl script and convert the timestamp from twitter to an easier format.

Is there an easier way to do this?

Answer 1

Skipping in MongoDB (as in a relational database) is very costly, particularly when the database collection grows to a very large size (as a twitter stream collection might). This is because Mongo has to literally sort and scan the entire collection, then iterate through but not return the number of documents you are skipping.

A better way to solve this problem is to use a query to “seek” to the right spot in your collection, and then return new results from that point. I’ll assume that your tweet documents have a timestamp field (a BSON date). When you render the page (or update it via AJAX), you will need to remember the latest date of what you’ve seen so far. Then you can query for new tweets like:

db.tweets.find({timestamp: {$gt: last_seen_timestamp}}).sort({timestamp: 1})

This will be efficient assuming you have an index on timestamp (or a composite index in which timestamp is the first field).