Question

I am currently using mongodb, and I see very bad performance of queries (It could take seconds).
The scenario is as following:

I have a documents of structure:

{_id:"xxx", userId:"yyy", a:1 ,b:2,  counter:1}    

In the test the:

"userId" value could be {1..200,000}
"a" values could be {1..30}
"b" values could be {1}

Thus my collection of maximun size will be 6,000,000
Currently there are two indexes defined for this collection : default _id and useId

The business logic queries for all user entries, and then updates one specific by incrementing the counter (the query update is written by “_id” ). Also if this is a new entity there is an insert query.

I’m running with mongo 1.8.2 on ubuntu with 8g ram

I have a master secondaries replications (all the mongo’s runs with local disk storage and in one subnetwork with tomcat server). Of course all the reads go to secondary and writes to master.
I didn’t tested sharding since i think that 6,000,000 is not a huge collection, isn’t it?

In addition i run jmetter test that generates 500 threads requests at a time with different userIds.

When i ran mongostat i see that the %locked is very high (about 70%)
after about 5-10 minutes of load i see that qw (queue for write) is 500 (as a number of my open connections)
When i stop the server it takes the mongo about 10-20 minutes to fulfill all the queued tasks

I’ve also ran db.serverStatus() and explain and the results looks fine.
when i run the db.currentOp() i see the queries that are waiting for ‘write’ lock
I could not have the output of currentOp to file to fully analyze it, because i executed the query from the commandline and had only the window buffer size. But from there i saw a lot of updates (by _id) that are waiting to write lock.

I’ll appreciate any ideas.

One more things: since each query pottentially will bring 30 documents I think there could be different moddeling as following:

{_id:"xxx", userId:"123", bs: [{b:1, cs[{c:1, cnt:1}, {c:2, cnt:1}}, {{b:2 cs: [{c:1, cnt:1}]}}]

But when i tryed this modelling, i could not increment the counter, I simply didn’t find the right way to do that. I can do insert and push bud can not update
for the following query :

db.coll.update({userId:"123", "bs.b":1, "bs.cs.c":1}, {"bs.cs.cnt" : {$inc : 1})

I have an error about the illegal ‘dot’ in the query

I’m pretty stacked by now. Waiting for some good ideas

Thanks a lot
Julia

Answer 1

MongoDB has a global write lock. This means that only one of your updates can proceed at a time.

The db.serverStatus() command can help you diagnose issues with the global write lock.

Here are some things you can try:

1) Make sure you’re using mongodb 2.0. It has better concurrency than older versions. 2.2 will have better concurrency yet.

2) Queue your writes so that they are asynchronous, and perform them all using a single thread. This might help with concurrency, because generally only one thread will be attempting to use the global write lock at a time.

3) If you’re using the latest version, and you can’t make your writes single threaded, then consider sharding. Sharding is for much more than just size; it’s at least as important for write concurrency. If you shard, then each segment will run in its own process with its own global write lock. This will allow the whole system to process more writes.