Question

I have double A[B_ROWS][B_COLUMNS]; in C API I used stufflike:

MPI_Isend(&A[low_bound][0], (upper_bound - low_bound) * A_COLUMNS, MPI_DOUBLE, i, MASTER_TO_SLAVE_TAG + 2, MPI_COMM_WORLD, &request);

and

 MPI_Recv(&A[low_bound][0], (upper_bound - low_bound) * A_COLUMNS, MPI_DOUBLE, 0, MASTER_TO_SLAVE_TAG + 2, MPI_COMM_WORLD, &status);

Now with boost::mpi I try:

world.isend(i, TO_SLAVE_TAG + 2, &A[low_bound][0], (upper_bound - low_bound) * A_COLUMNS);

and

world.recv(0, TO_SLAVE_TAG + 2, &A[low_bound][0], (upper_bound - low_bound) * A_COLUMNS);

but my app constantly fails with stuff like:

rank 1 in job 10  master_39934   caused collective abort of all ranks
  exit status of rank 1: killed by signal 11

which means seg fault, please note that original C app worked as needed and all that I currently changed was use of api – not any logic around.

So what is the correct way of sending 2d C style arrays over boost::mpi?

Answer 1

Assuming that my blind guess is right, and what you typed above is accurate, the size of A has nothing to do with A_COLUMNS (instead, A has B_COLUMNS). If so, the code below will fix that kind of “out of sync” error:

template<typename World, typename T>
void isend( World& w, int dest, int tag, T const* t, size_t n = 1) {
  world.isend(dest, tag, &t, n);
}
template<typename World, typename T, size_t aSize>
void isend( World& w, int dest, int tag, T const (*arr1)[aSize], size_t n = 1) {
  world.isend(dest, tag, &(*arr)[0], n*aSize);
}
template<typename World, typename T, size_t aSize, size_t bSize>
void isend( World& w, int dest, int tag, T const (*arr2)[aSize][bSize], size_t n = 1) {
  world.isend(dest, tag, &(*arr)[0][0], n*aSize*bSize);
}

template<typename World, typename T>
void recv( World& w, int dest, int tag, T* t, size_t n = 1) {
  world.recv(dest, tag, &t, n);
}
template<typename World, typename T, size_t aSize>
void recv( World& w, int dest, int tag, T (*arr1)[aSize], size_t n = 1) {
  world.recv(dest, tag, &(*arr)[0], n*aSize);
}
template<typename World, typename T, size_t aSize, size_t bSize>
void recv( World& w, int dest, int tag, T (*arr2)[aSize][bSize], size_t n = 1) {
  world.recv(dest, tag, &(*arr)[0][0], n*aSize*bSize);
}

the above code will, for one and two dimensional arrays, figure out how many copies of T you really want to send, instead of you having to maintain it manually.

It even works for slices, like &A[low_bound], upper_bound-lower_bound.

One thing you might want to be careful of is blowing past the end of your array. It is easily possible that your C code blew past the end of arrays, but there wasn’t anything important there, so it survived. In the C++ code, you could have an object there, and you die instead of survive.

Another approach might be to write a function that takes both upper and lower bounds for a slice, like this:

template<typename World, typename T, size_t N>
void isend_slice( World& w, int dest, int tag, T const (&t)[N], size_t start=0, size_t end=N ) {
  Assert( end <= N && start < end );
  isend(world, dest, tag, &(t[start]), end-start);
}
template<typename World, typename T, size_t N>
void recv_slice( World& w, int dest, int tag, T (&t)[N], size_t start=0, size_t end=N ) {
  Assert( end <= N && start < end );
  recv(world, dest, tag, &(t[start]), end-start);
}

in this case, you pass an array directly, and then say where you want to start and end reading. The advantage is that I check that the array actually has the data to send, or room for the data to arrive.

(These two functions rely on the functions above)

In a distributed situation, you’d want to produce a logging mechanism for your Asserts that is descriptive.

Here is sample use of the above code:

int array[10];
int array2[10][10];
isend(world, dest, tag+0, &int(7)); // tag is an int
isend(world, dest, tag+1, &array); // tag+1 is a 10 int array
isend(world, dest, tag+2, &array2); // tag+2 is a 100 int array
isend(world, dest, tag+1, &(array2[5])); // tag+1 is a 10 int array
isend_slice(world, tag+3, 0, array2, 7, 11); // asserts, but its a 40 int array

and ditto for recv.