Question

Background/Intent:

So I’m going to create an event tracker from scratch and have a couple of ideas on how to do this but I’m unsure of the best way to proceed with the database side of things. One thing I am interested in doing is allowing these events to be completely dynamic, but at the same time to allow for reporting on relational event counters.

For example, all countries broken down by operating systems. The desired effect would be:

1. US # of events
   • iOS – # of events that occured in US
   • Android – # of events that occured in US
2. CA # of events
   • iOS – # of events that occured in CA
   • Android – # of events that occured in CA
3. etc.

My intent is to be able to accept these event names like so:

/?country=US&os=iOS&device=iPhone&color=blue&carrier=Sprint&city=orlando&state=FL&randomParam=123&randomParam2=456&randomParam3=789

Which means in order to do the relational counters for something like the above I would potentially be incrementing 100+ counters per request.

Assume there will be 10+ million of the above requests per day.

I want to keep things completely dynamic in terms of the event names being tracked and I also want to do it in such a manner that the lookups on the data remains super quick. As such I have been looking into using redis or mongodb for this.

Questions:

1. Is there a better way to do this then counters while keeping the fields dynamic?

2. Provided this was all in one document (structured like a tree), would using the $inc operator in mongodb to increment 100+ counters at the same time in one operation be viable and not slow? The upside here being I can retrieve all of the statistics for one ‘campaign’ quickly in a single query.

3. Would this be better suited to redis and to do a zincrby for all of the applicable counters for the event?

Thanks

Answer 1

Depending on how your key structure is laid out I would recommend pipelining the zincr commands. You have an easy “commit” trigger – the request. If you were to iterate over your parameters and zincr each key, then at the end of the request pass the execute command it will be very fast. I’ve implemented a system like you describe as both a cgi and a Django app. I set up a key structure along the lines of this:

YYYY-MM-DD:HH:MM -> sorted set

And was able to process Something like 150000-200000 increments per second on the redis side with a single process which should be plenty for your described scenario. This key structure allows me to grab data based on windows of time. I also added an expire to the keys to avoid writing a db cleanup process. I then had a cronjob that would do set operations to “roll-up” stats in to hourly, daily, and weekly using variants of the aforementioned key pattern. I bring these ideas up as they are ways you can take advantage of the built in capabilities of Redis to make the reporting side simpler. There are other ways of doing it but this pattern seems to work well.

As noted by eyossi the global lock can be a real problem with systems that do concurrent writes and reads. If you are writing this as a real time system the concurrency may well be an issue. If it is an “end if day” log parsing system then it would not likely trigger the contention unless you run multiple instances of the parser or reports at the time of input. With regards to keeping reads fast In Redis, I would consider setting up a read only redis instance slaved off of the main one. If you put it on the server running the report and point the reporting process at it it should be very quick to generate the reports.

Depending on your available memory, data set size, and whether you store any other type of data in the redis instance you might consider running a 32bit redis server to keep the memory usage down. A 32b instance should be able to keep a lot of this type of data in a small chunk of memory, but if running the normal 64 bit Redis isn’t taking too much memory feel free to use it. As always test your own usage patterns to validate