Question

I have a large scipy.sparse.csc_matrix and would like to normalize it. That is subtract the column mean from each element and divide by the column standard deviation (std)i.

scipy.sparse.csc_matrix has a .mean() but is there an efficient way to compute the variance or std?

Answer 1

You can calculate the variance yourself using the mean, with the following formula:

E[X^2] - (E[X])^2

E[X] stands for the mean. So to calculate E[X^2] you would have to square the csc_matrix and then use the mean function. To get (E[X])^2 you simply need to square the result of the mean function obtained using the normal input.