Question

I am new to Parallel Programming. I am trying to multiply two matrices. I have partitioned the problem as follows:

Let the operation be mat3 = mat1 x mat2
I am broadcasting the mat2 to all the processes in the communicator, and cutting out strips of rows of the mat1 and scattering them to the processes in the commnicator group. After all processes has the entire mat2 and the corresponding strips of mat1, they multiply the strip with mat2 and then I am using the gather operation with the local results of the process, and accumulate the entire result in the root process.

I wanted to know if there is a better problem partitioning to multiply two matrix in a general purpose computer.

My implementation is in OpenMPI.

Answer 1

There is a variety of algorithms in the literature about Matrix Multiplication that can be extend to the MPI paradigm. For example:

> 1Dsystolic [1] 
> 2D-systolic, Cannon’s algorithm [2];
> Fox’s algorithm [3];
> Berntsen’s algorithm [4];
> DNS algorithm [5].

If you ignore the matrix proprieties (sparse ect), it basically resumes on how the data its distributed among the processes to minimize the synchronization and the load unbalance (the amount of work distributed among each process).

In this recent work you can see two different data distribution approach and the comparison between them.

papers:

[1] Golub G.H and Van C.H L., “Matrix Computations.”,Johns Hopkins University Press, 1989.
[2] Whaley R. C., Petitet A., Dongarra J. J., “Automated empirical optimizations of software and the ATLAS project” Parallel Computing 27, 1.2 (2001), 3.35.
[3] Fox G. C., Otto S. W., and Hey A. J. G., “Matrix algorithms on a hypercube I:
Matrix multiplication”,Parallel Computing, vol. 4, pp. 17-31. 1987.
[4] Berntsen J.,“Communication efficient matrix multiplication on hypercubes, Parallel Computing”, vol. 12, pp. 335-342, 1989.
[5] Ranka S. and Sahni S., “Hypercube Algorithms for Image Processing and Pattern Recognition”, Springer- Verlag, New York, NY, 1990.