Question

I am seeing a behavior with my mongos where the the C# driver SendMessage takes substantially longer when things get queued up on my one of my shards and mongos has to forward calls to that shard. Shouldn’t the SendMessage be independent from how busy the shard is and I should be able to queue things based on my network connection speed and not how busy the shard is?

Answer 1

Shouldn’t the SendMessage be independent from how busy the shard is and I should be able to queue things based on my network connection speed and not how busy the shard is?

MongoDB does not use a buffer, but the operating system/network stack has one for all network traffic (and by socket/connection).

What you are really talking about is the shard server’s network buffer. You’re saying “hey it doesn’t matter if you’re busy, just take this request”. However, the network buffer is not infinitely long. At some point, the buffer will be full and that connection will simply stop accepting more data until it clears up.

If you are at the point where you are sending more operations/data than a single shard can handle, you have a limited number of options:

1. More hardware (disk/mem) => more shards => less load on a single shard
2. Double-check your shard key distribution (though it is almost impossible to change)
3. Try some manual chunking to alleviate the load

Unfortunately, none of these are easy solutions. MongoDB is great in that it provides the ability to shard data. But it has no automated features to handle “hot” nodes. If know which chunks are hot then you can split them and balancer will move them around but there is no automatic process which does this for you, if the data in chunk doesn’t change.