Question

I have an OpenMP parallelized program that looks like that:

[...]
#pragma omp parallel
{
//initialize threads

#pragma omp for
for(...)
  {
  //Work is done here

  }

}

Now I’m adding MPI support. What I will need is a thread that handles the communication, in my case, calls GatherAll all the time and fills/empties a linked list for receiving/sending data from the other processes. That thread should send/receive until a flag is set. So right now there is no MPI stuff in the example, my question is about the implementation of that routine in OpenMP.
How do I implement such a thread? For example, I tried to introduce a single directive here:

[...]
int kill=0
#pragma omp parallel shared(kill)
{
//initialize threads
#pragma omp single nowait
 {
  while(!kill)
   send_receive(); 
 }
#pragma omp for
for(...)
  {
  //Work is done here

  }
kill=1

} 

but in this case the program gets stuck because the implicit barrier after the for-loop waits for the thread in the while-loop above.

Thank you, rugermini.

Answer 1

You could try adding a nowait clause to your single construct:

EDIT: responding to the first comment

If you enable nested parallelism for OpenMP, you might be able to achieve what you want by making two levels of parallelism. In the top level, you have two concurrent parallel sections, one for the MPI communications, the other for local computation. This last section can itself be parallelized, which gives you a second level of parallelisation. Only threads executing this level will be affected by barriers in it.

#include <iostream>
#include <omp.h>

int main()
{
  int kill = 0;
#pragma omp parallel sections
  {
#pragma omp section
    {
      while (kill == 0){
        /* manage MPI communications */
      }
    }

#pragma omp section
    {
#pragma omp parallel
#pragma omp for
      for (int i = 0; i < 10000 ; ++i) {
        /* your workload */
      }
      kill = 1;
    }
  }
}

However, you must be aware that your code is going to break if you don’t have at least two threads, which means you’re breaking the assumption that the sequential and parallelized versions of the code should do the same thing.

It would be much cleaner to wrap your OpenMP kernel inside a more global MPI communication scheme (potentially using asynchronous communications to overlap communications with computations).