We’re working with an average sized (50GB) data set in MongoDB and are attempting to add a third node to our replica set (making it primary-secondary-secondary). Unfortunately, when we bring the nodes up (with the appropriate command line arguments associating them with our replica set), the nodes never exit the RECOVERING stage.
Looking at the logs, it seems as though the nodes ditch all of their data as soon as the recovery completes and start syncing again.
We’re using version 2.0.3 on all of the nodes and have tried adding the third node from both a “clean” (empty db) state as well as a bootstrapped state (using mongodump to take a snapshot of the primary database and mongorestore’ing that snapshot into the new node), each failing.
We’ve observed this recurring phenomenon over the past 24 hours and any input/guidance would be appreciated!
It’s hard to be certain without looking at the logs, but it sounds like you’re hitting a known issue in MongoDB 2.0.3. Check out http://jira.mongodb.org/browse/SERVER-5177 . The problem is fixed in 2.0.4, which has an available release candidate.