Question

I’m building an application that stores lots of data per user (possibly in gigabytes).

Something like a request log, so lets say you have the following fields for every record:

customer_id
date
hostname
environment
pid
ip
user_agent
account_id
user_id
module
action
id
response code
response time (range)

and possibly some more.

The good thing is that the usage will be mostly write only, but when there are reads
I’d like to be able to answer then quickly in near real time.

Another prediction about the usage pattern is that most of the time people will be looking at the most recent data,
and infrequently query for the past, aggregate etc, so my guess is that the working set will be much smaller then
the whole database, i.e. recent data for most users and ranges of history for some users that are doing analytics right now.
for the later case I suppose its ok for first query to be slower until it gets the range into memory.

But the problem is that Im not quite sure how to effectively index the data.

The start of the index is clear, its customer_id and date. but the rest can be
used in any combination and I can’t predict the most common ones, at least not with any degree of certainty.

We are currently prototyping this with mongo. Is there a way to do it in mongo (storage/cpu/cost) effectively?

The only thing that comes to mind is to try to predict a couple of frequent queries and index them and just massively shard the data
and ensure that each customer’s data is spread evenly over the shards to allow fast table scan over just the ‘customer, date’ index for the rest
of the queries.

P.S. I’m also open to suggestions about db alternatives.

Answer 1

with this limited number of fields, you could potentially just have an index on each of them, or perhaps in combination with customer_id. MongoDB is clever enough to pick the fastest index for each case then. If you can fit your whole data set in memory (a few GB is not a lot of data!), then this all really doesn’t matter.

You’re saying you have a GB per user, but that still means you can have an index on the fields as there are only about a dozen. And with that much data, you want sharding anyway at some point soon.

cheers,
Derick