Question

I’m considering using MongoDB or CouchDB on a project that needs to maintain historical records. But I’m not sure how difficult it will be to store historical data in these databases.

For example, in his book “Developing Time-Oriented Database Applications in SQL,” Richard Snodgrass points out tools for retrieving the state of data as of a particular instant, and he points out how to create schemas that allow for robust data manipulation (i.e. data manipulation that makes invalid data entry difficult).

Are there tools or libraries out there that make it easier to query, manipulate, or define temporal/historical structures for key-value stores?

edit:

Note that from what I hear, the ‘version’ data that CouchDB stores is erased during normal use, and since I would need to maintain historical data, I don’t think that’s a viable solution.

P.S. Here’s a similar question that was never answered: key-value-store-for-time-series-data

Answer 1

There are a couple options if you wanted to store the data in MongoDB. You could just store each version as a separate document, as then you can query to get the object at a certain time, the object at all times, objects over ranges of time, etc. Each document would look something like:

{
    object : whatever,
    date : new Date()
}

You could store all the versions of a document in the document itself, as mikeal suggested, using updates to push the object itself into a history array. In Mongo, this would look like:

db.foo.update({object: obj._id}, {$push : {history : {date : new Date(), object : obj}}})

// make changes to obj
...

db.foo.update({object: obj._id}, {$push : {history : {date : new Date(), object : obj}}})

A cooler (I think) and more space-efficient way, although less time-efficient, might be to store a history in the object itself about what changed in the object at each time. Then you could replay the history to build the object at a certain time. For instance, you could have:

{
    object : startingObj,
    history : [
        { date : d1, addField : { x : 3 } },
        { date : d2, changeField : { z : 7 } },
        { date : d3, removeField : "x" },
        ...
    ]
}

Then, if you wanted to see what the object looked like between time d2 and d3, you could take the startingObj, add the field x with the value 3, set the field z to the value of 7, and that would be the object at that time.

Whenever the object changed, you could atomically push actions to the history array:

db.foo.update({object : startingObj}, {$push : {history : {date : new Date(), removeField : "x"}}})