Home/ Questions/Q 9197429
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-17T22:04:22+00:00

If a mongo node is offline for too long and the oplog wraps before

  • 0

If a mongo node is offline for too long and the oplog wraps before it comes back up then it can get stuck in a stale state and require manual intervention. How can I recognise that state from the replica set status document? Will it stick in state 3, which is also used by nodes in maintenance mode and presumably by nodes that can catch up? If so, how can I tell the difference?

From http://docs.mongodb.org/manual/reference/replica-status/:

Number State
0      Starting up, phase 1 (parsing configuration)
1      Primary
2      Secondary
3      Recovering (initial syncing, post-rollback, stale members)
4      Fatal error
5      Starting up, phase 2 (forking threads)
6      Unknown state (the set has never connected to the member)
7      Arbiter
8      Down
9      Rollback
10     Removed
Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    It will be in state 3, Recovering. To recognize the stale state specifically you need to look for the errmsg field. When stale, the secondary in question will have an errmsg like this:

    "errmsg" : "error RS102 too stale to catch up"

    In terms of a full output, it would look something like this:

     rs.status()
{
        "set" : "testReplSet",
        "date" : ISODate("2013-01-29T01:39:38Z"),
        "myState" : 1,
        "members" : [
                {
                        "_id" : 0,
                        "name" : "hostname:31000",
                        "health" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY",
                        "uptime" : 507,
                        "optime" : Timestamp(1359423456000, 893),
                        "optimeDate" : ISODate("2013-01-29T01:37:36Z"),
                        "self" : true
                },
                {
                        "_id" : 1,
                        "name" : "hostname:31001",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 483,
                        "optime" : Timestamp(1359423456000, 893),
                        "optimeDate" : ISODate("2013-01-29T01:37:36Z"),
                        "lastHeartbeat" : ISODate("2013-01-29T01:39:37Z"),
                        "pingMs" : 0
                },
                {
                        "_id" : 2,
                        "name" : "hostname:31002",
                        "health" : 1,
                        "state" : 3,
                        "stateStr" : "RECOVERING",
                        "uptime" : 4,
                        "optime" : Timestamp(1359423087000, 1),
                        "optimeDate" : ISODate("2013-01-29T01:31:27Z"),
                        "lastHeartbeat" : ISODate("2013-01-29T01:39:38Z"),
                        "pingMs" : 0,
                        "errmsg" : "error RS102 too stale to catch up"
                }
        ],
        "ok" : 1
}

    And finally, a code snippet to print out the error only, if it exists, from the shell:

    rs.status().members.forEach(function printError(rsmember){if (rsmember.errmsg){print(rsmember.errmsg)}})
    • 0