Question

I am writing an application which is recording some ‘basic’ stats — page views, and unique visitors. I don’t like the idea of storing every single view, so have thought about storing totals with a hour/day resolution. For example, like this:

Tuesday    500 views   200 unique visitors
Wednesday  400 views   210 unique visitors
Thursday   800 views   420 unique visitors

Now, I want to be able to query this data set on chosen time periods — ie, for a week. Calculating views is easy enough: just addition. However, adding unique visitors will not give the correct answer, since a visitor may have visited on multiple days.

So my question is how do I determine or estimate unique visitors for any time period without storing each individual hit. Is this even possible? Google Analytics reports these values — surely they don’t store every single hit and query the data set for every time period!?

I can’t seem to find any useful information on the net about this. My initial instinct is that I would need to store 2 sets of values with different resolutions (ie day and half-day), and somehow interpolate these for all possible time ranges. I’ve been playing with the maths, but can’t get anything to work. Do you think I may be on to something, or on the wrong track?

Thanks,
Brendon.

Answer 1

If you are OK with approximations, I think tom10 is onto something, but his notion of random subsample is not the right one or needs a clarification. If I have a visitor that comes on day1 and day2, but is sampled only on day2, that is going to introduce a bias in the estimation. What I would do is to store full information for a random subsample of users (let’s say, all users whose hash(id)%100 == 1). Then you do the full calculations on the sampled data and multiply by 100. Yes tom10 said about just that, but there are two differences: he said “for example” sample based on the ID and I say that’s the only way you should sample because you are interested in unique visitors. If you were interested in unique IPs or unique ZIP codes or whatever you would sample accordingly. The quality of the estimation can be assessed using the normal approximation to the binomial if your sample is big enough. Beyond this, you can try and use a model of user loyalty, like you observe that over 2 days 10% of visitors visit on both days, over three days 11% of visitors visit twice and 5% visit once and so forth up to a maximum number of day. These numbers unfortunately can depend on time of the week, season and even modeling those, loyalty changes over time as the user base matures, changes in composition and the service changes as well, so any model needs to be re-estimated. My guess is that in 99% of practical situations you’d be better served by the sampling technique.