Question

I need to store sparse matrix data.
Size of data is 10^6 10^4 columns.
In each column I store a vector of 0’s except of few values where it’s true.

Then I need to sum over columns in each matrix, and multiply each row with a scalar.
I tried dictionaries, but they fail when I need to sum, and multiply.

What would You use?

PS. numpy.zeros is too small

Answer 1

How about two dicts? Assuming this is the matrix (x for True):

   0  1  2  3  4  5  6  7
0  x     x        x 
1     x
2                       x
3              x
4
5
6        x        x
7

You’d only need to store

rows = {0: [0, 2, 5], 1: [1], 2: [7], 3: [4], 6: [2, 5]}

You could easily transform this into

columns = {0: [0], 1: [1], 2: [0, 6], 4: [3], 5: [0, 6], 7: [2]}

using something like

columns = {}
for row in rows:
    for column in rows[row]:
        columns.setdefault(column, []).append(row)

and then sum over columns (sum(1 for x in column[2])) or over rows and multiply the result with whatever you want.