Question

I have a 200gb data base on a sharded four node cluster and and I would like to drop the databse and delet all the files associated to it from the node. I am connecting to my mongos and call dropDatabase on it. The system comes back with ok but if call show dbs, it will show the database again and shows that it is still occupying the 200gb. What I am doing wrong?

Answer 1

I think you are running into this issue:

https://jira.mongodb.org/browse/SERVER-4804

In most cases it seems like the database is in fact removed but the mongos still reports it as being there. You can verify it is gone by either trying to use the DB and getting an error or by logging into the shards directly and checking.

The bug refers to issues with dropping databases while a migration is happening. You can workaround the cause of the issue by doing something like this (sub in your own dbname):

mongos> use config
switched to db config

// 1. stop the balancer
mongos> db.settings.update({_id: "balancer"}, {$set: {stopped: true}}, true)

// 2. wait for in-progress migrations to finish, this may take a few seconds
mongos> while (db.locks.findOne({_id: "balancer", state: {$ne: 0}}) != null) { sleep(1000); }

// 3. now you can safely drop the database
mongos> use <dbname>
switched to db <dbname>
mongos> db.dropDatabase()
{ "dropped" : "<dbname>", "ok" : 1 }

You may want to run the flushRouterConfig on the mongoses to refresh the config info:

mongos> use config
switched to db config
mongos> var mongoses = db.mongos.find()
mongos> while (mongoses.hasNext()) { new Mongo(mongoses.next()._id).getDB("admin").runCommand({flushRouterConfig: 1}) }
{ "flushed" : true, "ok" : 1 }

Of course, the real fix will only come along when the fix is committed – looks like it is targeted for 2.1

If you are in a broken state, you can try this, but it is tricky:

To “reset” the sharding metadata to recover from this issue, please try to do the following

First, stop the balancer (as above) and wait for migrations to finish (also as above)
Next, ensure there is no activity from the app servers on the database in question

Now, ensure that there are no collection entries in config.collections for namespaces beginning with “TestCollection.” If so, remove those entries through a mongos:

mongos> use config
mongos> db.collections.find({_id: /^TestCollection\./})
// if any records found, delete them
mongos> db.collections.remove({_id: /^TestCollection\./})

Next up, ensure there is no database entry in config.databases for “TestCollection”, and if so, remove it through mongos:

mongos> use config
switched to db config
mongos> db.databases.find({_id: "TestCollection"})
// if any records found, delete them
mongos> db.databases.remove({_id: "TestCollection"})

Now, ensure there are no entries in config.chunks for any namespaces in the database (like the default test namespace). If there are any, remove through mongos:

mongos> use config
switched to db config
mongos> db.chunks.find({ns: /^test\./})
// if any records found, delete them
mongos> db.chunks.remove({ns: /^test\./})

Then, flushRouterConfig on all mongoses:

mongos> use config
switched to db config
mongos> var mongoses = db.mongos.find()
mongos> while (mongoses.hasNext()) { new Mongo(mongoses.next()._id).getDB("admin").runCommand({flushRouterConfig: 1}) }
{ "flushed" : true, "ok" : 1 }
...

Finally, manually connect to each shard primary, and drop the database on the shards (not all shards may have the database, but it’s best to be thorough and issue the dropDatabase() call on all

Regarding in-progress migrations, you can use this snippet:

// 2. wait for in-progress migrations to finish, this may take a few seconds
mongos> while (db.locks.findOne({_id: "balancer", state: {$ne: 0}}) != null) { sleep(1000); }

When done, don’t forget to reenable the balancer:

mongos> use config
switched to db config

mongos> db.settings.update({_id: "balancer"}, {$set: {stopped: false}}, true)