Question

Consider something like this:

---

typedef struct TS { 
    double a,b,c; 
} S; 

... 
S x,y; 
... 
MPI_Allreduce(&x, &y, 3, MPI_DOUBLE, MPI_SUM, MPI_COMM_WORLD); 

---

Is the above code completely portable (without using MPI_Type_struct and all; all variables in the structure are assumed to be of the same type)? Also in the case when different hardware on various nodes is used?

Thanks in advance,
Jac

Answer 1

Hristo Iliev’s completely right; the C standard allows arbitrary padding between the fields. So there’s no guarantee that this is the same memory layout as an array of three doubles, and your reduce could give you garbage.

So there’s two different approaches you could take here. One is to ignore the problem, as most C compilers probably will treat this as an array of three contiguous doubles. I normally wouldn’t mention this at all as even an option except that in this case it’s so easy to test the assumption; in your code you can have

assert ( offsetof(S,b) == sizeof(double) );
assert ( offsetof(S,c) == 2*sizeof(double) );

and if your code proceeds through the asserts you’re good. (Note that this still doesn’t guarantee that an array of two of these structs is equivalent to an array of 6 contiguous doubles…)

The second way is to create the structure and reduce operation yourself to be safe. And really, it’s not too difficult, and then you know it’ll work, so that’s really the way to go; and then you can use that type for any other operations safely:

#include <stdio.h>
#include <stddef.h>
#include <mpi.h>

typedef struct TS {
    double a,b,c;
} S;

/* our reduction operation */
void sum_struct_ts(void *in, void *inout, int *len, MPI_Datatype *type){
    /* ignore type, just trust that it's our struct type */

    S *invals    = in;
    S *inoutvals = inout;

    for (int i=0; i<*len; i++) {
        inoutvals[i].a  += invals[i].a;
        inoutvals[i].b  += invals[i].b;
        inoutvals[i].c  += invals[i].c;
    }

    return;
}

void defineStruct(MPI_Datatype *tstype) {
    const int count = 3;
    int          blocklens[count];
    MPI_Datatype types[count];
    MPI_Aint     disps[count];

    for (int i=0; i < count; i++) {
        types[i] = MPI_DOUBLE;
        blocklens[i] = 1;
    }

    disps[0] = offsetof(S,a);
    disps[1] = offsetof(S,b);
    disps[2] = offsetof(S,c);

    MPI_Type_create_struct(count, blocklens, disps, types, tstype);
    MPI_Type_commit(tstype);
}

int main (int argc, char **argv) {

    int rank, size;
    MPI_Datatype structtype;
    MPI_Op       sumstruct;
    S   local, global;

    MPI_Init(&argc, &argv);
    MPI_Comm_size(MPI_COMM_WORLD, &size);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);

    defineStruct(&structtype);
    MPI_Op_create(sum_struct_ts, 1, &sumstruct);

    local.a = rank;
    local.b = 2*rank;
    local.c = 3*rank;

    MPI_Reduce(&local, &global, 1, structtype, sumstruct, 0, MPI_COMM_WORLD);

    if (rank == 0) {
        printf("global.a = %lf; expected %lf\n", global.a, 1.*size*(size-1)/2);
        printf("global.b = %lf; expected %lf\n", global.b, 2.*size*(size-1)/2);
        printf("global.c = %lf; expected %lf\n", global.c, 3.*size*(size-1)/2);
    }

    MPI_Finalize();
    return 0;
}

Running gives

$ mpicc -o foo foo.c -std=c99

$ mpirun -np 1 ./foo
global.a = 0.000000; expected 0.000000
global.b = 0.000000; expected 0.000000
global.c = 0.000000; expected 0.000000

$ mpirun -np 2 ./foo
global.a = 1.000000; expected 1.000000
global.b = 2.000000; expected 2.000000
global.c = 3.000000; expected 3.000000

$ mpirun -np 3 ./foo
global.a = 3.000000; expected 3.000000
global.b = 6.000000; expected 6.000000
global.c = 9.000000; expected 9.000000

$ mpirun -np 12 ./foo
global.a = 66.000000; expected 66.000000
global.b = 132.000000; expected 132.000000
global.c = 198.000000; expected 198.000000