Question

Imagine following situation:

There is a distributed key/value database stored on computer network. One central “main” computer that fetches request, and multiple child machines that store portions of data.
I.e. something like this:

main computer
     |
     +--child A
     +--child B
     +--child C
     .....

I.e. “star” topology.

Additional description:

1. Portions of database overlap, and several different versions of record with same “key” can be stored on several machines at same time.
2. Key is not guaranteed to exist on all machines or on specific machines.
3. “Children” do not synchronize data with each other.
4. Data is requested/read only via main computer, which must return most recent version of data for requested key.
5. Data is written only through children – they receive new values from several sources.
6. Data is never deleted.

Now the main problem:

With such structure, how do I determine which version is most recent?

I can think of two ways to deal with the problem:

1. Add timestamp for every record, when it is written into database vial child machine, use timestamp to determine version.
2. Use “revision number” or “write operation index” (issued by main computer, increments by one for every write operation) instead of timestamps.

However, both approaches are not perfect:
1st approach requires perfect clock synchronization for all machines, otherwise system will fail to deliver most recent record value.
2nd approach will cause every child to ask main machine for timestamp via network, which will introduce writing delays, plus main machine will have to be locked by mutex, so multithreading performance will suffer.

What is the better way to deal with this situation?
How do real clustered databases deal with this situation (most recent record version in cluster)?

Answer 1

Your statement that the first approach requires perfect clock synchronization is not correct.

You do not care about the absolute timestamps issued by a child, only the relative timestamps. So as long as the clocks advance at the same rate, they need not be synchronized; you can correct for the known offsets.

If the clocks on the children advance at different rates, then you must use a method which involves coordination (writing cannot be lock-free in the slow path). This is provable by contradiction, since obviously two children independently writing a value with time-records that cannot be related to each other will not let an outside observer determine which was written later.

However, you can do the coordination in parallel with the actual write: write to the child and, simultaneously, to an ordered log which allows a determination of which write happened first (you don’t need a ticket-type system like you seem to suggest if you’ve got a write log). So it doesn’t necessarily delay the process of writing at all!

Take a look at logical-timestamp key-value systems like Accumulo, an HBase alternative (currently in Apache-project incubation) – this is real world clustered database doing exactly what you’re asking for.