Question

I am trying to make sure that I understand what happens when I add a new Shard (Replica Set) to an existing Shard Cluster. When I add these new members and it sees that there is a new Shard Member available Mongo then starts to re-arrange the chunks so that it can take advantage of the new members correct? What sort of impacts to you get when this happens? As always I would assume you want to try and add these members as soon as you start to see unfavorable performance numbers (If other tuning options are not helping).

Just wanted to get a better understanding of what happens when you add a Shard when a cluster already exists.

Thanks,

S

Answer 1

When you add a shard to an existing cluster, it will automatically become the shard with the lowest number of chunks for every sharded collection. That means that it will be the default target for migrations (from the shard with the highest number of chunks) until things get more balanced. However, each shard primary (which is responsible for the migrations) can only take part in one migration at a time. As such, the balancing is going to take a while, especially if things are under load.

In terms of the migrations themselves, you are seeing them in your current cluster already, so that is how to judge their impact. You can view the recent migrations in the logs, or you can take a look at the changelog (a 10MB capped collection that contains the most recent migrations/splits etc.):

// connect to a mongos, switch to the config DB
use config
// look at the changelog
db.changelog.find()

In terms of what operations happen, well to move a chunk:

1. The documents that make up that chunk must be read into memory on the source shard if not already there (so a fairly standard read)
2. They are then sent to the destination shard (a fairly standard insert/write)
3. Finally, after the meta data has been updated, they are removed from the source shard.

Step 3 is a delete, which requires a write lock on the source shard, but it should be quite fast – the documents are already in memory from the migration.

One other impact of increasing the frequency of the migrations is that the shard version is going to be updated more frequently – in particular the major shard version (so that it has an up to date mapping of chunks to shards.

That means that you will see more logged messages about the mongos needing to refresh its config and update its shard version. It may also be a good idea to run the flushRouterConfig command before you kick off long running operations like Map/Reduce or findAndModify.

If your shards have periods of low usage, you will see the migrations happen more quickly, and you can also consider using the balancer window option to only run balancing during certain times if you do notice a significant impact.