Question

What is the best practice for running a database-query after any document in a collection become of certain age?

Let’s say this is a node.js web-system with mongoDB, with a collection of posts. After a new post is inserted, it should be updated with some data after 60 minutes.

Would a cron-job that checks all posts with (age < one hour) every minute or two be the best solution? What would be the least stressing solution if this system has >10.000 active users?

Answer 1

Some ideas:

• Create a second collection as a queue with a “time to update” field which would contain the time at which the source record needs to be updated. Index it, and scan through looking for values older than “now”.
• Include the field mentioned above in the original document and index it the same way
• You could just clear the value when done or reset it to the next 60 minutes depending on behavior (rather than inserting/deleting/inserting documents into the collection).
• By keeping the update-collection distinct, you have a better chance of always keeping the entire working set of queued updates in memory (compared to storing the update info in your posts).
• I’d kick off the update not as a web request to the same instance of Node but instead as a separate process so as to not block user-requests.

As to how you schedule it — that’s up to you and your architecture and what’s best for your system. There’s no right “best” answer, especially if you have multiple web servers or a sharded data system.

You might use a capped collection, although you’d run the risk of potentially losing records needing to be updated (although you’d gain performance)