Question

I have a large amount of data I need to store, and be able to generate reports on – each one representing an event on a website (we’re talking over 50 per second, so clearly older data will need to be aggregated).

I’m evaluating approaches to implementing this, obviously it needs to be reliable, and should be as easy to scale as possible. It should also be possible to generate reports from the data in a flexible and efficient way.

I’m hoping that some SOers have experience of such software and can make a recommendation, and/or point out the pitfalls.

Ideally I’d like to deploy this on EC2.

Answer 1

Wow. You are opening up a huge topic.

A few things right off the top of my head…

1. think carefully about your schema for inserts in the transactional part and reads in the reporting part, you may be best off keeping them separate if you have really large data volumes
2. look carefully at the latency that you can tolerate between real-time reporting on your transactions and aggregated reporting on your historical data. Maybe you should have a process which runs periodically and aggregates your transactions.
3. look carefully at any requirement which sees you reporting across your transactional and aggregated data, either in the same report or as a drill-down from one to the other
4. prototype with some meaningful queries and some realistic data volumes
5. get yourself a real production quality, enterprise ready database, i.e. Oracle / MSSQL
6. think about using someone else’s code/product for the reporting e.g. Crystal/BO / Cognos

as I say, huge topic. As I think of more I’ll continue adding to my list.

HTH and good luck