I have to develop a web frontend to visualize data (using d3.js).
The data is delivered by an external database.
Requirements:
- Frontend holds the data from the (at least) last two hours (one datum per system per minute).
- Clients (Browsers) connect to the Web-Frontend, click on the corresponding system and sees some graphs. They should be updated regularly.
- Also dynamic requests/graphs should be possible to request.
My approach:
-
FrontEnd (PHP App) to generate the pages (containing the data from the last two hours) and create a websocket (to update the data on the frontend via push (JSON))
-
Write an importer-utility which queries the external db (which might have huge response times) and write it into a local ‘buffer’ (could be a in-memory db like redis) and then notify my PHP-App to push the updates. (Or let the php-app check the db regularly)
-
Visualize the data with d3, update the graphs with the new data on push-message from the server (my php app)
Is this idea worth trying? Do you have any recommendations, readings and so on?
The idea sounds reasonable. One place to look at for inspiration, at least, is http://square.github.com/cubism/.
If you use graphite or cube in the place where you are considering redis, you would have something that gets you fairly close ‘out-of-the-box’.
Much of the consideration is around the data and the visualization. I would take into consideration if and how the data may pivot. In other words, will the user always see the same view, or could they see aggregate data “by machine”, “by geography”, and “by purpose”.