I am going to be writing some software in PHP to parse log files and aggregate the data then display them in graphs (like bar graphs, not vertices and edges).
Yeah, it’s basically business intelligence software which my company has an entire team for but apparently they don’t do a great job (10 minutes to load a page just doesn’t do it).
Here is what i have to do:
- Log files are data files which stores the raw data from a stats server we have setup running from our office (we send asynchronous calls to the stats server kinda like google analytics). It stores the data in csv format.
- write a script to parse the files and aggregate the data into a database (or i was thinking about redis)
- There will be millions and millions of things to aggregate so when displaying stats it must be fast
I know about OLAP for the DB, but if i want to go with redis do you think it would scale for large volumes of data? To parse the files do you think a PHP script would suffice or should i go with something faster like C/C++?
Basically i would like to get some interesting ideas about different ways to accomplish my task. It must be fast and scale.
Any ideas?
It sounds like at the scales you’re talking about, you need to separate the data aggregation and display. That is, you should have some process working to receive the log files when they’re generated, parse them and insert the data into the database; that will be a long, complicated task. Then when a user wants to display a graph of the data, they can make a request to the PHP server, which will pull the data from the database and construct the display they want. In this way, your parsing is separated from your display request (although it’s still serially dependent, your parsing can begin when the logfiles become available, and therefore, the lag involved in parsing them is hidden at display time).