Question

I have a few hundred network devices that check in to our server every 10 minutes. Each device has an embedded clock, counting the seconds and reporting elapsed seconds on every check in to the server.
So, sample data set looks like

CheckinTime               Runtime
2010-01-01 02:15:00.000   101500
2010-01-01 02:25:00.000   102100
2010-01-01 02:35:00.000   102700

etc.

If the device reboots, when it checks back into the server, it reports a runtime of 0.

What I’m trying to determine is some sort of quantifiable metric for the device’s “health”.

If a device has rebooted a lot in the past but has not rebooted in the last xx days, then it is considered healthy, compared to a device that has a big uptime except for the last xx days where it has repeatedly rebooted.
Also, a device that has been up for 30 days and just rebooted, shouldn’t be considered “distressed”, compared to a device that has continually rebooted every 24 hrs or so for the last xx days.

I’ve tried multiple ways of calculating the health, using a variety of metrics:
1. average # of reboots
2. max(uptime)
3. avg(uptime)
4. # of reboots in last 24 hrs
5. # of reboots in last 3 days
6. # of reboots in last 7 days
7. # of reboots in last 30 days

Each individual metric only accounts for one aspect of the device health, but doesn’t take into account the overall health compared to other devices or to its current state of health.

Any ideas would be GREATLY appreciated.

Answer 1

You could do something like Windows’ 7 reliability metric – start out at full health (say 10). Every hour / day / checkin cycle, increment the health by (10 - currenthealth)*incrementfactor). Every time the server goes down, subtract a certain percentage.

So, given a crashfactor of 20%/crash and an incrementfactor of 10%/day:

• If a device has rebooted a lot in the past but has not rebooted in the last 20 days will have a health of 8.6

• Big uptime except for the last 2 days where it has repeatedly rebooted 5 times will have a health of 4.1

• a device that has been up for 30 days and just rebooted will have a health of 8

• a device that has continually rebooted every 24 hrs or so for the last 10 days will have a health of 3.9

To run through an example:

Starting at 10
Day 1: no crash, new health = CurrentHealth + (10 - CurrentHealth)*.1 = 10
Day 2: One crash, new health = currenthealth - currentHealth*.2 = 8
But still increment every day so new health = 8 + (10 - 8)*.1 = 8.2
Day 3: No crash, new health = 8.4
Day 4: Two crashes, new health = 5.8