Question

I can’t seem to figure out how to send data between nodes instead of sending it to the root node and then sending it to all other nodes.

If I have N nodes each with an array created like so, where SIZE is the total number of nodes, and for the moment assume it’s a preprocessor constant (when possible, avoid malloc like the plague). Also, it goes without saying that rank is the current node’s rank.

int dummy [SIZE][5];
int i, n;

for (n = 0; n < SIZE; n++){
  for (i = 0; i <5; i++){
    if ( n == rank ){
      dummy [ n ][ i ] = 123;

This gives each node a nearly empty array of dimensions SIZE * 5, with only one row of the numbers 123. Now I want to take all these individual arrays and ‘merge’ them. The only thing I can think of is the below, but I am certain this will lead to a deadlock, even if I do bother checking to see if I source node does not equal target node:

for ( i = 0; i < SIZE; i++ ){
  for ( j = 0; j < SIZE; j++ ){
     MPI_Send ( &dummy [ i ], 5, MPI_INT, j, 123, MPI_COMM_WORLD );
  }
}

for ( i = 0; i < SIZE; i++ ){
  for ( j = 0; j < SIZE; j++ ){
     MPI_Recv ( &dummy [ j ], 5, MPI_INT, i, 123, MPI_COMM_WORLD );
  }
}

Can someone kindly provide me with some pseudocode as to how to tackle this problem. Cheers

Answer 1

This is a “gather” operation, and there’s an MPI collective, MPI_Gather(), which implements it if you want to gather all of the data on to one processor, and MPI_Allgather() gathers the data to all processors.

In this case, we want to do an ‘in-place’ gather – gathering into the same array we’re sending from. So this will work:

#include <stdio.h>
#include <stdlib.h>
#include <mpi.h>

#define SIZE 5

int main(int argc, char **argv) {

    int size, rank;
    MPI_Init(&argc, &argv);
    MPI_Comm_size(MPI_COMM_WORLD, &size);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);

    if (size != SIZE) {
        if (rank == 0) {
            fprintf(stderr,"Must run with %d ranks\n", SIZE);
        }
        MPI_Finalize();
        exit(1);
    }

    int dummy [SIZE][5];

    for (int i = 0; i <5; i++){
          dummy [ rank ][ i ] = 100 * rank;
    }

    MPI_Allgather(MPI_IN_PLACE, 5, MPI_INT,
                  dummy, 5, MPI_INT,
                  MPI_COMM_WORLD);

    if (rank == SIZE-1) {
        printf("Rank %d has dummy:\n", rank);
        for (int i=0; i<size; i++) {
            for (int j=0; j<5; j++) {
                printf("%3d ", dummy[i][j]);
            }
            printf("\n");
        }
    }

    MPI_Finalize();
    return 0;
}

Running gives:

$ mpicc -o allgather allgather.c -std=c99
$ mpirun -np 5 ./allgather
Rank 4 has dummy:
  0   0   0   0   0 
100 100 100 100 100 
200 200 200 200 200 
300 300 300 300 300 
400 400 400 400 400