For example, our application tracks animal movements and prices for a farm. To get the current stock count the simplest solution is to have a starting number, then add up all the movement in and out until we have a current number. But this is memory intensive and gets slower and slower as the number of movements grows year after year.
We don’t have the luxury of “freezing” a year so it can no longer accept changes, the system must be able to handle changes to movements at any point in time, then show the updated numbers in real time.
This is not just stock numbers; we have to track a large number of variables like this and write reports for each period (day, week, month, year) that include summary calculations based on these variables.
What is the most common, preferred, “best”, fastest, elegant way to handle data streams that cross multiple years for calculation and reporting purposes? How would the database design and the architecture relate in this scenario (i.e. would using an ORM be fine as long as the database schema was well designed?). The critical requirements here are optimal performance and real-time availability.
I have seen in large scale systems thus kind of work is split up into time slices, e.g. week, month, year aggregate tables. I am particularly interested if there is a common design pattern for solving this problem.
I would aggregate in the DB, as that’s usually something they are very good at.
Have a look at OLAP (vs OLTP) database design.