I have developed an application which allows users to enter measurements – these are stored in an Oracle database. Each measurement “session” could contain around 100 measurements. There could be around 100 measurement sessions in a “batch”, so that’s 10,000 measurements per batch. There could easily be around 1000 batches at some point, bringing the total number of measurements into the millions.
The problem is that calculations and statistics need to be performed on the measurements. It ranges from things like average measurements per batch to statistics across the last 6 months of measurements.
My question is: is there any way that I can make the process of calculating these statistics faster? Either through the types of queries I’m running or the structure of the database?
Thanks!
Unfortunately, as the number of calculations that needed to be carried out are not limited to just a few, I could not calculate them at the end of every measurement session.
In the end the queries did not take so long – around 3 minutes for calculating all the stats. For end users this was still an unacceptably long time to wait, BUT the good thing was that the stats did not necessarily have to be completely up to date.
Therefore I used a materialized view to take a ‘snapshot’ of the stats, and set it to update itself every morning at 2am. Then, when the user requested the stats from the materialized view it was instant!