Question

Let’s say you have a 3d array A[I][J][K], but you want to permute it into B[J][K][I].

This problem is similar to, but different from, the 2-d array transpose problem discussed here:
Can you transpose array when sending using MPI_Type_create_subarray?.

The MPI standard provides examples of multi-dimensional array operations with user-defined datatypes, but not this particular case: http://www.mpi-forum.org/docs/mpi-11-html/node61.html

Answer 1

The way to do this is with MPI_TYPE_VECTOR and MPI_TYPE_CREATE_HVECTOR, but the devil is in the details.

/* data[I][J][K] is I by J by K (stored in array 'dim_sizes[] = {I, J, K}'
   and we want permuted[J][K][I] */

/* new innermost dimension is I items, strided across the old JK face*/
MPI_Type_vector(dim_sizes[0], 1, dim_sizes[1]*dim_sizes[2], 
    MPI_DOUBLE, &one_d);
MPI_Type_commit(&one_d);

/* new middle dimenson is K items, strided over the K row, which isn't
 * actually a stride in this case.  We use hvector here because we 
 * operate directly in terms of array items */
MPI_Type_create_hvector(dim_sizes[2], 1, sizeof(double), one_d, &two_d);
MPI_Type_commit(&two_d);

/* new outermost dimension is J items, strided over the old J row */
MPI_Type_create_hvector(dim_sizes[1], 1, dim_sizes[2]*sizeof(double), two_d,
    &transposed_type);
MPI_Type_commit(&transposed_type);

Now you can feed transposed_type to your send/receive call or make it your MPI file view.