Question

We’re using MongoDB 2.2.0 at work. The DB contains about 51GB of data (at the moment) and I’d like to do some analytics on the user data that we’ve collected so far. Problem is, it’s the live machine and we can’t afford another slave at the moment. I know MongoDB has a read lock which may affect any writes that happen especially with complex queries. Is there a way to tell MongoDB to treat my (particular) query with the lowest priority?

Answer 1

In MongoDB reads and writes do affect each other. Read locks are shared, but read locks block write locks from being acquired and of course no other reads or writes are happening while a write lock is held. MongoDB operations yield periodically to keep other threads waiting for locks from starving. You can read more about the details of that here.

What does that mean for your use case? Because there is no way to tell MongoDB to access the data without a read lock, nor is there a way to prioritize the requests (at least not yet) whether the reads significantly affect the performance of your writes depends on how much “headroom” you have available while write activity is going on.

One suggestion I can make is when figuring out how to run analytics, rather than scanning the entire data set (i.e. doing an aggregation query over all historical data) try running smaller aggregation queries on short time slices. This will accomplish two things:

1. reads jobs will be shorter lived and therefore will finish quicker, this will give you a chance to assess what impact the queries have on your “live” performance.
2. you won’t be pulling all old data into RAM at once – by spacing out these analytical queries over time you will minimize the impact it will have on current write performance.

Depending on what it is you can’t afford about getting another server – you might consider getting a short lived AWS instance which may be not very powerful but would be available to run a long analytical query against a copy of your data set. Just be careful when making it a copy of your data – doing a full sync off of the production system will place a heavy load on it (more effective way would be to use a recent backup/file snapshot to resume from).