Question

I have a hybrid code with MPI/OpenMP. I want to know what is the time spent for particular function, let’s say A, for every MPI process. This function is called inside the OpenMP do/for loops also in a very complicated way by various functions on top of it (i.e. some other functions let’s say B and C may be calling A which also might be inside the OpenMP do/for loops). I was planning to do it as follows:

double A()
{
 time1 = MPI_Wtime();
 //compute result...
 //Note: inside this function there is no OpenMP or MPI calls...
 //      just pure computation of results...
 time2 = MPI_Wtime();
 printf("myRank=%d timeSpent=%f\n", myRank, (time2-time1));
 return result;
}

Would the sum of all the times per every MPI process be the total time spent for this function by that MPI process? If not please can you show me how to get it correctly, thanks!

Answer 1

We don’t want to reinvent the wheel and we don’t want to reinvent the MPI profiler. That would be hard.

There are very powerful tools available from the manufactures of many cluster systems. For example Cray machines usually come with CrayPat which spits out magic.

Additionally there is free software such as this http://mpip.sourceforge.net/