Question

I’m just starting to approach this problem, I want to to allow users to arbitrarily select ranges and filters that allow them to graph large data sets (realistically it should never be more than 10 million data points) on a web page. I use elasticsearch as the method of storing and aggregating the data, along with redis for keeping track of summary data, and d3.js is my graphing library.

My thoughts on the best solution is to have precalculated summaries in different groupings that can be used to graph from. So if the data points exist over several years, I can have groupings by month and day (which I would be doing anyway), but then by groupings of say half day, quarter day, hour, half hour, etc. And then before I query for graph data I do a quick calculation to see which of these groupings will give me some ideal number of data points (say 1000).

Is this a reasonable way to approach the problem? Is there a better way?

Answer 1

You should reconsider the amount of data…

1. Even in desktop plotting apps it is uncommon to show that many points per plot – e.g. origin prints a warning that it will show only a subset for performance reasons. you could for example throw away every 3rd point to make them less.
2. You should give the user the ability to zoom in or navigate around to explore the data, like pagination alike style …
3. Grouping or faceting how it is called in Lucene community is of course possible with that many documents but be sure you have enough RAM+CPU