Question

Ever since I have added a new database into the mongodb, it stopped syncing replicaSet secondary instances, i.e. database name appear when running show dbs yet appear as (empty)

There is a repeating error in the log file at the secondary which also appear in

"errmsg" : "syncTail: ...

below is the output of rs.Status() on primary

PRIMARY> rs.status()
{
        "set" : "contoso_db_set",
        "date" : ISODate("2012-11-01T13:05:22Z"),
        "myState" : 1,
        "syncingTo" : "dbuse1d.int.contoso.com:27017",
        "members" : [
                {
                        "_id" : 0,
                        "name" : "dbuse1a.int.contoso.com:27017",
                        "health" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY",
                        "optime" : {
                                "t" : 1351775119000,
                                "i" : 2
                        },
                        "optimeDate" : ISODate("2012-11-01T13:05:19Z"),
                        "self" : true
                },
                {
                        "_id" : 1,
                        "name" : "dbuse1d.int.contoso.com:27017",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 4108139,
                        "optime" : {
                                "t" : 1351405977000,
                                "i" : 12
                        },
                        "optimeDate" : ISODate("2012-10-28T06:32:57Z"),
                        "lastHeartbeat" : ISODate("2012-11-01T13:05:21Z"),
                        "pingMs" : 1,
                        "errmsg" : "syncTail: 10068 invalid operator: $oid, syncing: { ts: Timestamp 1351576230000|1, h: -2878874165043062831, op: \"i\", ns: \"new_contoso_db.accounts\", o: { _id: { $oid: \"4f79a1d1d4941d3755000000\" }, delegation: [ \"nE/UhsnmZ1BCCB+tiiS8fjjNwkxbND5PwESsaXeuaJw=\""
                },
                {
                        "_id" : 2,
                        "name" : "dbuse1a.int.contoso.com:8083",
                        "health" : 1,
                        "state" : 7,
                        "stateStr" : "ARBITER",
                        "uptime" : 10671267,
                        "optime" : {
                                "t" : 0,
                                "i" : 0
                        },
                        "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
                        "lastHeartbeat" : ISODate("2012-11-01T13:05:21Z"),
                        "pingMs" : 0
                }
        ],
        "ok" : 1
}
PRIMARY> 

Answer 1

The solution I have found was to delete the entire database from secondary

# rm -rf /data/db
# mkdir -p /data/db

And then restarting mongo and setting up the the replicaSet.

See more at Mongo’s doc

What to do on a RS102 sync error

If one of your members has been offline and is now too far behind to
catch up, you will need to resync. There are a number of ways to do
this.
Perform a full resync. If you stop the failed mongod, delete all data in the dbpath (including subdirectories), and restart it, it will
automatically resynchronize itself. Obviously it would be better/safer
to back up the data first. If disk space is adequate, simply move it
to a backup location on the machine if appropriate. Resyncing may take
a long time if the database is huge or the network slow – even
idealized one terabyte of data would require three hours to transmit
over gigabit ethernet.*