Question

Regarding my previous question ( How to quickly build large scale analytics server?
) I’m now feeding my analytics data to MongoDB – Every event (with bunch of metadata) gets their own document in views collection.

However, now I’ve hit the next roadblock: Now that inserts are done and analytics data is flowing, what would be the path of least resistance to do runs against that data? The idea is that once that data is sharded, the specific views would run mapReduces (say all events with specific ID one month back).

So, my question is: As I’m quite new with MongoDB, what are the steps I would need to do to get those mapReduces as fast as possible? Should I structure the raw data differently or is the one document per event correct? Are there Mongo specific tricks that I can do to make things flow faster when running against dataset that gets millions of inserts per day?

I would prefer to keep my technology stack as simple as possible (Node.js + MongoDB), so I’d prefer if things could be done without introducing extra technology (like say Hadoop).

Example document of a event:

{
    id: 'abc',
    ip: '1.1.1.1',
    type: 'event1',
    timestamp: 1234,
    metadata: {
        client: 'client1'
    }
}

All main aggregations will be ID centric, analysing events in said ID, the most used being get all events with said ID for the last month. Minor aggregations would be grouping things with metadata (how many percents used client1 vs. client2 etc.). All the aggregations will be system defined, so users can’t set them by themselves, at this point at least. Thus, as far as I understand, the sharding should be done through ID, as big majority of the aggregations will be ID centric? Also, this should mean that the most recent events on any given ID are always in memory as Mongo keeps the latest stuff in memory and only dumps the overflow to disk.

Also, real time is not a requirement. Although, ofc it would be nice. 😛

Edit: Added example data

Edit: Title should have been “…per day” not “…per page” + more spec about the aggregation sets

Answer 1

To expand on the other answer with my own flavour; map reduce is a solid option for a lot of your processing. Make no mistake, map reduce will be a vital core of your analytics if you do it fully.

The best sort of MR is an incremental one which builds things like archival statistics on certain events and people so as to shrink the overall database size and working set size of pulling out all that old, dusty data.

As for realtime usage, I have found (from many dicussions on the subject in early days of MongoDB in the Google User group) the best way is to pre-aggregate your data so that a simple linear query like: db.some_preaggregated_data.find({date: {$gt, $lt}}) will get your results. This eases the sharding of the data and also the ranging over it, not to mention your working set size. Overall giving a much more performant operation.

I would recommend one thing:- don’t go into the aggregation framework or complex queries if you really expect this to scale fully in realtime. It will start to create too much work on such a hugely expansive data set. You will need the full lock, working set etc on your side using simple find() queries to satisfy your needs across a big table format normally.