Question

On every request of a webpage I have to multiply a matrix by a vector. The matrix is symmetric and (probably) sparse, with dimensions of approximately 500×500. The vector is a column taken out of another matrix with dimensions of approximately 500×100000. Now my question is how I store the matrices and execute the calculation in an effective way. I would like to do the calculations with php and store the matrices with mysql or on harddrive, but I assume there are better tools for this kind of task.

If you need to know anything further, don’t heasitate to ask!

Answer 1

OK, so we got a real corner-case here:

• 500×100,000=50,000,000 elements. Depending on the data type, this
  translates to ca. 50M-400M of in-memory representation if we ignore
  sparsity and address by index. The matrix being symmetric reduces this to 25-200M
• A non-zero density of 11% means, that an overhead of 9 Elements for
  sparse adressing is a break even – not really convincing for small
  data types, but might be an option for 64bit data types.

Working from this gives strong argument to a two-tier solution:

• A (shared) “data store server” (MySQL being the one option you
  mentioned)
• and a (maybe PHP) frontend.

Using PHP for a 500×500 matrix multiplication doesn’t seem to be a very efficient way to go: Loop overhead is quite high in PHP, so with so little done inside the loop, you might be wasting a bit of performance.

I recommend to go for a slightly different solution: Write your own backend (maybe in C or C#), that

• On startup reads the matrix into memory
• Listens on a (local) socket for commands, e.g. “multiply”
• Executes those commands on the matrix and given data and returns the result

and feed it via PHP. This way you get the processing speed of a compiled language for the core function, while having the simplicity and web-centricity of PHP for the frontend.