Question

I’ve got multiple websites, where each website has visitors that “trigger” multiple events I want to track. I have a log of those events, from all websites, each event is filled with the website-id, the event-name and the user-id that did the event (for the sake of simplicity, let’s say that’s it).

The requirements:

1. Be able to get, per website-id and event-name, how many unique visitors got it.
2. This should support also date range (distinct unique visitors on the range).

I was thinking of creating a collection per “website-id” with the following data model (as example):

collection ev_{websiteId}:
[
    {
        _id: "error"
        dailyStats: [
            {
                _id: 20121005 <-- (yyyyMMdd int, should be indexed!)
                hits: 5
                users: [ 
                         {
                            _id: 1, <-- should be indexed!
                            hits: 1
                         }, 
                         {
                            _id: 2
                            hits: 3
                         },
                         {
                            _id: 3,
                            hits: 1
                         }
                ]
            },
            {
                _id: 20121004 
                hits: 8
                users: [ 
                         {
                            _id: 1,
                            hits: 2
                         }, 
                         {
                            _id: 2
                            hits: 3
                         },
                         {
                            _id: 3,
                            hits: 3
                         }
                ]
            },
        ]
    },
    {
        _id: "pageViews"
        dailyStats: [
            {
                _id: 20121005 
                hits: 500
                users: [ 
                         {
                            _id: 1, 
                            hits: 100
                         }, 
                         {
                            _id: 2
                            hits: 300
                         },
                         {
                            _id: 3,
                            hits: 100
                         }
                ]
            },
            {
                _id: 20121004
                hits: 800
                users: [ 
                         {
                            _id: 1, 
                            hits: 200
                         }, 
                         {
                            _id: 2
                            hits: 300
                         },
                         {
                            _id: 3,
                            hits: 300
                         }
                ]
            },
        ]
    },
]

I’m using the _id to hold the event-id.
I’m using dailyStats._id to hold when it happened (an integer in yyyyMMdd format).
I’m using dailySattes.users._id to represent a user’s unique-id hash.

In order to get the unique users, I should basically be able to run (mapreduce?) distinct count number of items in the array(s), per the given date range (I will convert the date range to yyyyMMdd).

My questions:

1. does this data model makes sense to you? I’m concerned about scalability of this model over time (if I’ve got a lot of daily unique visitors in some client, it make cause a huge document).
   I was thinking of deleting dailyStats documents by _id < [date as yyyyMMdd]. This way I can keep my documents size to a sane number, but still, there are limits here.
2. Is there an easy way to run “upsert” that will also create the dailyStats if not already created, add the user, if not already created and increment “hits” property for both?
3. what about map-reduce? how would you approach it (need to run distinct on the users._id for all subdocuments in the given date range)? is there an easier way with the new aggregation framework?

btw – another option to solve unique visitors is using Redis Bitmaps but I am not sure it’s worth holding multiple data storage (maintenance-wise).

Answer 1

Few comments on the current above architecture.

I’m slightly worried as you’ve pointed out about the scalability and how much pre-aggregation you’re really doing.

Most of the Mongo instances I’ve worked with when doing metrics have similar cases to what you pointed out but you really seem to be relying heavily on doing updates to a single document and upserting various parts of it which is going to slow down and potentially cause a bit of locking..

I might suggest a different path, one that Mongo even suggests when talking with them about doing metrics. Seeing as you already have a structure that you’re looking to do I’d create something along the lines of:

{
  "_id":"20121005_siteKey_page",
  "hits":512,
  "users":[
   {
     "uid":5, 
     "hits":512,
   }
}

This way you are limiting your document sizes to something that is going to be reasonable to do quick upserts on. From here you can do mapreduce jobs in batches to further extend out what you’re looking to see.

It also depends on your end goal, are you looking to provide realtime metrics? What sort of granularity are you attemtping to get? Redis Maps may be something you want to at least look at: Great article here.

Regardless it is a fun problem to solve 🙂

Hope this has helped!