Question

I followed Jonathan’s code from here ( MPI_Bcast a dynamic 2d array ) to MPI_Bcast a dynamically allocated 2d array of structs. The struct is as follows:

typedef struct {
   char num[MAX];
   int neg;
} bignum; 

Since this is not a native datatype, I used MPI_Type_create_struct to commit this datatype.
I omit the code here because it works on another project I was doing.

The method I used to dynamically create (“contiguous?”) the array is by calling:

 bignum **matrix = creatematrix(DIMMAX,DIMMAX);

which is implemented as:

 bignum ** creatematrix (int num_rows, int num_cols){
   int i;
   bignum *one_d = (bignum*)malloc (num_rows * num_cols * sizeof (bignum));
   bignum **two_d = (bignum**)malloc (num_rows * sizeof (bignum *));

   for (i = 0; i < num_rows; i++)
   two_d[i] = &(one_d[i*num_cols]);//one_d + (i * num_cols);

   return two_d;
 }

Now I’d get input and store it inside matrix and call MPI_Bcast as:

MPI_Bcast(&(matrix[0][0]), DIMMAX*DIMMAX, Bignum, 0, MPI_COMM_WORLD);

There seems to be no segmentation fault, but the problem is that only the first row (matrix[0]) is broadcasted. All other ranks beside the root have data only for the first row. Other rows remain untouched.

It looks like there’s something more than allocating a contiguous memory block. Is there anything I missed causing broadcast to be unsuccessful?

EDIT:

I fumbled on a weird workaround, using nested structs. Can anyone explain why this method works while the one above doesn’t?

typedef struct {
   char num[MAX];
   int neg;
} bignum;

typedef struct {
   bignum elements[DIMMAX];
} row;

int main(/*int argc,char *argv[]*/){
  int dim, my_rank, comm_sz;
  int i,j,k; //iterators
  bignum global_sum = {{0},0};
  row *matrix = (row*)malloc(sizeof(row)*DIMMAX);
  row *tempmatrix = (row*)malloc(sizeof(row)*DIMMAX);
  MPI_Init(NULL,NULL);
  MPI_Comm_size(MPI_COMM_WORLD, &comm_sz);
  MPI_Comm_rank(MPI_COMM_WORLD, &my_rank);

  //CREATE DERIVED DATATYPE Bignum
  MPI_Datatype Bignum;  
  MPI_Datatype type[4] = {MPI_LB, MPI_CHAR, MPI_INT, MPI_UB};
  int blocklen[4] = {1,MAX,1,1};
  MPI_Aint disp[4];
  //get offsets 
  MPI_Get_address(&global_sum, disp);
  MPI_Get_address(&global_sum.num, disp+1); 
  MPI_Get_address(&global_sum.neg, disp+2);
  //MPI_Get_address(&local_sum+1, disp+3);
  int base;
  base = disp[0];
  for (i=0;i<3;i++){
      disp[i] -= base;  
  }
  disp[3] = disp[2]+4;  //int
  MPI_Type_create_struct(4,blocklen,disp,type,&Bignum);
  MPI_Type_commit(&Bignum);

  //CREATE DERIVED DATATYPE BignumRow
  MPI_Datatype BignumRow;   
  MPI_Datatype type2[3] = {MPI_LB, Bignum, MPI_UB};
  int blocklen2[3] = {1,DIMMAX,1};  
  MPI_Aint disp2[3] = {0,0,DIMMAX*(MAX+4)};
  MPI_Type_create_struct(3,blocklen2,disp2,type2,&BignumRow);
  MPI_Type_commit(&BignumRow);

  MPI_Bcast(&dim, 1, MPI_INTEGER, 0, MPI_COMM_WORLD);
  MPI_Bcast(matrix, dim, BignumRow, 0, MPI_COMM_WORLD);

Answer 1

This seems to come up a lot: If you pass around multi-dimensional arrays with MPI in C, you sort of have to think of a multi-dimensional array as an array of arrays. I think in your first example you’ve gotten lucky that matrix[0][0] (due to the consecutive mallocs?) sits at the beginning of a large block of memory (and hence, no seg fault).

In the second (working) example, you’ve described the memory addresses of not just the beginning of the memory region (matrix[0][0]), but also all the internal pointers as well.