I need to design a real time statistics system for an online advertising project. I would need to store counters for number of impressions of a url, number of clicks, number of keywords displayed per impression etc. The amount of traffic can be assumed to be around 15000 impressions per second, but should be scalable to > 100000 impressions per second. The database operations are obviously highly write intensive, requiring fast,concurrent writes. What sort of database should I use and how would I go about designing the database architecture?
I am a real rookie in this field and am not really sure how to proceed. I dont have too much experience with NoSql databases, and I am not sure if operations of this scale can be achieved with a relational database like MySql or MS SQL Server.
Any help will be greatly appreciated!
Thanks.
On average 4-years old desktop PC running Windows, ESENT NoSQL is capable of 70k inserts/second (for small records). And it’s surely OK doing fast, concurrent writes. It even has the feature “escrow update column” that allows you to update int32 column values without locking a record, i.e. concurrently.
The DB architecture only depends on the exact data you’re going to collect.
P.S. Your task sounds like a hard to me. One problem is scalability: you better decide early if you’ll going to scale horizontally, i.e. by adding servers. If yes, it’ll affect the architecture a lot. Also, how you’ll obtain that amount of data? A web server is not capable of processing 100k requests/sec.