Question

Anyone have a good reference for how to do a multivariate ordinary linear regression without saving the input data (and get the R-squared of the result). The use case is a data set with too many rows to store. The regression can be obtained by accumulating x[i]*x[j] and y * x[i], and then doing the matrix math from there, but I can’t find a similar formula to get the statistics when I’m done (R-squared for starters). Thanks.

Answer 1

I don’t have a good reference, but the way I’d approach it is to
expand out the sum-of-squared expressions, and write them
in terms of the expectations that you are accumulating.

• I use <.> to indicate averaging over rows of data,
  so that <y> is the average of the y-values,
  and so on

• at any point we can obtain the regression coefficients a[i] and b
  from the matrix <x[i]*x[j]> and the vector <y*x[i]> as you indicated in your question

• below I’ll use sum_i{ a[i]*x[i] } to indicate a sum over the components
  that comprise the independent variables.
• Let N be the number of data rows used

A way to compute the explained mean-squared deviation is:

SS_reg/N = < (f -<y> )^2 >    

         = < ( sum_i {a[i]*x[i] } + b - <y> )^2 > 
         = < sum_i { a[i]^2*x[i]^2}  +b^2 +<y>^2 +sum_i{ 2*b*a[i]*x[i]}-2*<y>* sum_i{a[i]*x[i]}-2*b*<y> >
         = sum_i { a[i]^2*<x[i]*x[i]> } +
           b^2 +
           <y>^2 + 
           2*b*sum_i{a[i]*<x[i]>} -
           2*<y>*sum_i{ a[i]*<x[i]>} -
           2*b*<y>

You already maintain <x[i]*x[i]> as the diagonal elements of the matrix for
deriving the regression coefficients.
You will also need to maintain the averages of the independent variables
(<x[i]> for each i) as well as for the dependent variable (<y>)

Similar expansions can carried out for either the total or residual mean squared
errors, and then used to compute the R^2 value.