Question

I have an application that stores a vector of structs. These structs hold information about each GPU on a system like memory and giga-flop/s. There are a different number of GPUs on each system.

I have a program that runs on multiple machines at once and I need to collect this data. I am very new to MPI but am able to use MPI_Gather() for the most part, however I would like to know how to gather/receive these dynamically sized vectors.

class MachineData
{
    unsigned long hostMemory;
    long cpuCores;
    int cudaDevices;
    public:
    std::vector<NviInfo> nviVec; 
    std::vector<AmdInfo> amdVec;
    ...
};

struct AmdInfo
{
    int platformID;
    int deviceID;
    cl_device_id device;
    long gpuMem;
    float sgflops;
    double dgflops;
};

Each machine in a cluster populates its instance of MachineData. I want to gather each of these instances, but I am unsure how to approach gathering nviVec and amdVec since their length varies on each machine.

Answer 1

You can use MPI_GATHERV in combination with MPI_GATHER to accomplish that. MPI_GATHERV is the variable version of MPI_GATHER and it allows for the root rank to gather differt number of elements from each sending process. But in order for the root rank to specify these numbers it has to know how many elements each rank is holding. This could be achieved using simple single element MPI_GATHER before that. Something like this:

// To keep things simple: root is fixed to be rank 0 and MPI_COMM_WORLD is used

// Number of MPI processes and current rank
int size, rank;
MPI_Comm_size(MPI_COMM_WORLD, &size);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);

int *counts = new int[size];
int nelements = (int)vector.size();
// Each process tells the root how many elements it holds
MPI_Gather(&nelements, 1, MPI_INT, counts, 1, MPI_INT, 0, MPI_COMM_WORLD);

// Displacements in the receive buffer for MPI_GATHERV
int *disps = new int[size];
// Displacement for the first chunk of data - 0
for (int i = 0; i < size; i++)
   disps[i] = (i > 0) ? (disps[i-1] + counts[i-1]) : 0;

// Place to hold the gathered data
// Allocate at root only
type *alldata = NULL;
if (rank == 0)
  // disps[size-1]+counts[size-1] == total number of elements
  alldata = new int[disps[size-1]+counts[size-1]];
// Collect everything into the root
MPI_Gatherv(vectordata, nelements, datatype,
            alldata, counts, disps, datatype, 0, MPI_COMM_WORLD);

You should also register MPI derived datatype (datatype in the code above) for the structures (binary sends will work but won’t be portable and will not work in heterogeneous setups).