I’ve been developing with Django during the last year or so and really
enjoy it. But sometimes I find that the ORM is a bit of a
straitjacket. All the data that I shuffle back and forth to the
database would easily fit into 1GB of RAM. Even if the project grew a
couple of orders of magnitude it would still fit into 1GB.
I’d like a solution where my application only needs to read from disk
at startup but writes to disk implicitly as I update my objects. I
don’t care to much about any speed increase this might give me. What
I’m really after is the added flexibility. If I have a problem that
would fit nicely with a linked list or a tree or some other
data structure, I shouldn’t have to graft that onto a relational
database.
Python would be nice but other languages are fine. I’m in the exploratory
phase on this one. I want to get a feel for what solutions are
out there. When googling this question I got a lot of hits related to
different Nosql projects. But Nosql, as I understand it, is all about
what you do when you outgrow relational databases because you’ve got too
much data. I’m really at the other end of the spectrum. I’ve got so
little data that a relational database is actually overkill.
Object databases is an
other thing that came up when googling this question, which reminded
me of Zope and ZODB. I dabbled a bit with Zope an eon ago and really
disliked it. But reading up a bit on object databases made me think that it
might what I’m looking for. Then again, their general failure to
attract users makes me suspicious. Object databases have been around
for a very long time and still haven’t caught on. I guess
that means there’s something wrong with them?
If you are looking for “storing data-structures in-memory” and “backing up them to disk“, you are really looking for a persistent cache system and Redis fits the bill perfectly.
If you want to use django, there is an awesome cache system built-in and that is pluggable to redis backend via Redis-Cache project. More over the data-structures accomodated by redis have one-to-one mapping to python data-structures, and so it is pretty seamless.
I am not so sure if skipping the concept of database itself, is a good idea. Database provides so much power in terms of aggregating, annotating, relations etc, all within acceptable performance levels until you hit real large scale.
Perhaps another idea would be to to use SQLite in-memory database. SQLite is so ubiquitous these days, it has disappeared into the infrastructure. It is built in, Android apps, iphone ones and has support from all standard libraries. It is also an awesome software developed and tested so well, it is very hard to make any case against using it.