Ok, so I’ve been doing a bit of research into NoSQL databases, and they seem to be the right option for what I need. The problem is however, that a lot of these databases, if not most of them are reading to/writing from RAM, as opposed to disk. That’s great when you have plenty of server resources or don’t expect massive data blocks – but I think I should prepare for the worst.
What I expect to receive from these data sources is anywhere from 25KB to 150KB per query – yup – up to 150KB for a single key value. The average user will produce anywhere from 500 to 5000 of these keys and they can grow infinitely (but will probably stop somewhere in that 5000 range). If you quickly do the calculations (most of the data will be on the higher end of 25-150, so I’ll use 100KB as an “average”, most users will probably produce 2000-3000 queries): 100KB*3000 – that’s 300MB per user! An insane amount of data when you start getting a decent userbase. So, ultimately I’ll probably throw away most of the data in the queries so it is no more than 1KB or so, but that will still far surpass most RAM capabilities.
So I think what I’m looking for is a solution that will store data to disk, and cache objects in RAM.. But I’m open to all solutions! Let me know what you guys think. I would love to keep this thing running fast…
Edit:
Wording it slightly differently as to be useful to a passerby:
If one is looking to maximize performance but handle large dataloads in a NoSQL database, what would be the recommended NoSQL database? I would think it would be one which stores data to disk, but this can compromise performance significantly. Is there a “best of both worlds” solution out there? It is important to note I assume, that these records would not be modified once they were submitted, only read from (but maybe not even that often).
I’ve been looking into Redis for such a task, because it looks very clean to manage – however it runs entirely in RAM, thus requires small data blocks, or multiple servers running multiple instances at once.. Which is something I don’t have access to.
First of all, I think when you say most you’ve seen store data in RAM, you refer to in memory Key/Value data stores like Redis or Memcached.
But there’s more than that. Before closing the discussion on in-memory NoSQL options, I should say that you are right. Memory fills up quite easily and you would need tons of it, judging from your requirements. So in-memory options should be discarded (not they’re not useful, but not not in this specific situation).
My proposal is MongoDb. Does what you need: stores data on disk, caches stuff in-memory (as much as it can).
However, you need some powerful data storage options (SSD is what you should think about) so it can handle your data throughput needs. I’ve tested Mongo, but with far less data.
I was looking for over 1 million elements collections, with value sizes ranging from 5Kb to 50Kb.
I was mostly interested in read speeds. I should also mention write speeds, which I tested, and must say that they are impressive. One million 20Kb inserts in a few minutes (on a small server – quad core, 8GB of RAM, VMware VM).
Getting back to read speeds, I was looking for semi-concurrent queries that would give me under 50msec read times for around 100 concurrent users.
With some help from the MongoDb team I managed to get close to those times, but then I got into something else and had to drop my research (temporarily, I hope to resume it soon). There are far more things to look into, as speeds for aggregates, map/reduce, etc.
I can say that query times on the server were super fast and all the overhead was added by BSON serialization/deserialization and transport over the network.
So, for you Mongo would be appropriate, but you have to back it up with some good hardware.
You should really install it and test it in your specific situation and draw your conclusions from your own tests.
If you’re going to do it and your client is .NET, then you should use their official driver. Otherwise, there are plenty others listed here: http://www.mongodb.org/display/DOCS/Drivers.
A good intro on Mongo features and how to use them can be found here: http://www.mongodb.org/display/DOCS/Developer+Zone. Granted, their documentation is not as good as the one for RavenDb (another NOSQL solution I’ve tested, but not nearly as fast) but you can get good support here or on Google Groups.