Question

I’ve been working on an MPI project in which the slaves all send data back to the master. For some reason the master will only receive the data if I do 2 consecutive Sends in a row. This is very odd and I think it is causing some other weird problems I am getting. Any idea what would cause this? I think the first send is sending some kind of junk data or something. The sends are the exact same line of code though.

EDIT: Code below…

if (numProcs > 0)
    MPI_Barrier( MPI_COMM_WORLD ) ; //only wait if there are other processes to wait for

if (rank != 0)
{
    MPI_Send(handArray, 10, MPI_DOUBLE, 0, TAG_HAND, MPI_COMM_WORLD);
    MPI_Send(handArray, 10, MPI_DOUBLE, 0, TAG_HAND, MPI_COMM_WORLD);
}
//8. After the main loop the master process receives and sums together the hand counts array
//   from each slave process.
else
{
    int activeProcs = numProcs - 1;
    getHandsFromSlaves( activeProcs, handArray );

then the master proceeds to print some data…

Here is the getHands FromSlaves method. Please note I have also tried using blocking calls for this as well with the same problems.

void getHandsFromSlaves( int& activeCount, double handTotals[10] ){

static MPI_Request request;
static int msgBuff, recvFlag;
static double handBuff[10];
MPI_Status status;

while (activeCount > 0)
{
    if( request )
    {
        // Already listening for a message

        // Test to see if message has been received
        MPI_Test( &request, &recvFlag, &status );
        //cout << "TAG: " << status.MPI_TAG << " SOURCE: "<< status.MPI_SOURCE  << " ERROR: " << status.MPI_ERROR << endl;
        if( recvFlag )
        {
            // Message received
            if( status.MPI_TAG == TAG_HAND )
            {
                cout << "Hand Received!" << endl;

                for(int m = 0; m < 10; ++m)
                {
                    handTotals[m] += handBuff[m];
                }

                activeCount--;
            }
            else
            {
                //error report... what happened?
                cout << "TAG: " << status.MPI_TAG << " SOURCE: "<< status.MPI_SOURCE  << " ERROR: " << status.MPI_ERROR << endl;
            }

            // Reset the request handle
            request = 0;
        }
    }

    if( !request && activeCount > 0 )
        // Start listening again
        MPI_Irecv(&handBuff, 10, MPI_DOUBLE, MPI_ANY_SOURCE, MPI_ANY_TAG, MPI_COMM_WORLD, &request);
}
}

Answer 1

Well, you’re probably trying to process one too many messages because your request variable is undefined on entering your getHandsFromSlaves() routine. Since on enter, request is almost certainly non-zero, you immediately try to MPI_Test for a message something even though you haven’t posted an Irecv.

In fact, there’s a lot of really strange things about the code excerpt posted here. Why are the local variables static? Why would you implement your own busywait on MPI_Test() instead of using MPI_Wait()? Why are you using non-blocking receives at all if you’re not doing anything useful between receives? And indeed, if you’re just summing up all of the arrays anyway, why are you doing individual point-to-point receives at all instead of doing an MPI_Reduce()?

The following much shorter code seeems to do what you’re trying to do above:

#include <stdio.h>
#include <mpi.h>


int main (int argc, char **argv) {

    int rank, numProcs;

    MPI_Init(&argc, &argv);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &numProcs);
    double handArray[10];
    double handTotal[10];

    for (int i=0; i<10; i++)
        handArray[i] = rank + i;

    if (rank == 0)  // Since apparently rank 0 doesn't do anything
    {
        for (int i=0; i<10; i++)
            handArray[i] = 0;
    }

    MPI_Reduce(handArray, handTotal, 10, MPI_DOUBLE, MPI_SUM, 0, MPI_COMM_WORLD);

    if (rank == 0) {
        printf("Hand Totals= \n");
        for (int i=0; i<10; i++)
            printf(" %lf ", handTotal[i]);
        printf("\n");
    }

    MPI_Finalize();
}