Question

I have asked this question at various places, including the OpenMPI mailing list. So far, I’ve had no luck. So, I will appreciate, if someone can help, irrespective of how dumb the question may seem.

Problem:

Consider two processes A and B running on two different processors. No other
processes run. Process A has two messages to send, M1 (size: Large, tag T1) and
M2 (1 byte, tag T2). The process B uses MPI_Probe to selectively filter and
receive messages. Is it guaranteed that process B can still receive message M2,
when it does not MPI_Recv message M1? Consider that process A sent M1 first.
All processes are single-threaded.

Also, does OpenMPI ensure that two different messages from the same source
bearing the same tag can be received, in any order?

Basically, I want to understand if any role is played by OpenMPI’s system buffers: If one message manages to fill it up, then another message from the same source, but bearing another tag may never be received. Correct?

Thanks a lot for any ideas.

Devendra

Answer 1

I replied to you on the Open MPI users list, but will also post it here, just in case someone else find it useful. I believe Jeff Squyres has already answered your question on the Open MPI mailing list, or at least hinted on the possible problem. MPI messages are received in the order they were sent but only within the specific (tag, communicator) tuple. This basically means:

• within the same communicator you can receive messages out of order if they carry different tags;
• messages that carry the same tag can be received out of order if they were communicated in different contexts (communicators).

But here is the catch: you cannot receive a message if the send operation has not been posted yet. If you have two consecutive send operations, you must make sure that the first one would not block forever. The standard MPI send operation MPI_Send could be implemented in various ways (the standard doesn’t say exactly how), but in most MPI implementations it behaves like buffered send for very small messages and like synchronous send for larger messages. If you have the following two calls in your sender process:

MPI_Send(largedata, largecount, MPI_INT, dest, tag1, MPI_COMM_WORLD);
MPI_Send(smalldata, smallcount, MPI_INT, dest, tag2, MPI_COMM_WORLD);

it could happen that the first MPI_Send would actually behave as a synchronous one, i.e. it would not return unless the matching receive operation was posted on the receiver’s side. Suppose that your receiver code is:

MPI_Probe(src, tag2, MPI_COMM_WORLD, &status);
MPI_Recv(largedata, largecount, MPI_INT, src, tag1, MPI_COMM_WORLD, &status);

This would most likely deadlock because MPI_Probe is a blocking call, i.e. it would not return until a matching send was posted, i.e. the second MPI_Send would have to execute, which would only happen after the first send has returned, but it would not happen unless the MPI_Recv in the receiver is executed… I guess you get the idea

To prevent the deadlock you could modify the sender’s code to use a non-blocking send:

MPI_Request req;
MPI_Isend(largedata, largecount, MPI_INT, dest, tag1, MPI_COMM_WORLD, &req);
MPI_Send(smalldata, smallcount, MPI_INT, dest, tag2, MPI_COMM_WORLD);
MPI_Wait(&req, MPI_STATUS_IGNORE);

Using non-blocking operation the send call returns immediately and the operation continues in the background, so the second send would get executed immediately after that. Now there would be two pending messages and they can be received in any order since they carry different tags.