Question

I’m trying to run an mpi program using MPI_Comm_Spawn. I spawn 1 worker program, and then call MPI_reduce in both programs, to add up some results. For some reason, the application hangs at MPI_Comm_spawn, then aborts after a minute. The spawned process only gets to its code segment where its calls MPI_reduce after this happens. The application then continues to hang, and then gives some more errors in the command prompt. What should happen is that both the spawned and master programs reach the MPI_Reduce call, and the master program gets a sum, and outputs that sum.

Here’s the output, I’ve put a <> where its MPI’s output, and not my own

world size = 1   
About to call MPI_Comm_spawn with 2 workers...   
parent result is 3.141668952    
numDarts for child: 500000000  
argv[1] = 500000000  
<>MPI Application rank 0 killed before MPI_Finalize() with signal 11  
spawned process got result: 3.141668952  
Spawned process about to send message back to parent  
<>piworker: Rank 1:0: MPI_Finalize: IBV connection to 0 on card 0 is broken
<>piworker: Rank 1:0: MPI_Finalize: ibv_poll_cq(): bad status 12  
<>piworker: Rank 1:0: MPI_Finalize: self n93 peer n93 (rank: 0)  
<>piworker: Rank 1:0: MPI_Finalize: error message: transport retry exceeded error

Here’s the master program’s code:

#include "mpi.h"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include "globals.h"


int randSign();
double randFloat();
double dboard();


int main(int argc, char *argv[])
{
    int world_size, flag;
    MPI_Comm everyone;           /* intercommunicator */
    char worker_program[100];
    int universe_size;

    //  MPI_Comm_get_attr(MPI_COMM_WORLD, MPI_UNIVERSE_SIZE, &universe_size, &flag);
    //   printf("universe size: %i\n", universe_size);

    int numDarts = 1000000000;
    int numWorkers = 2;

    char* args[1];
    if(argc >= 2)
    {
      numWorkers = atoi(argv[1]);
    }
      if(argc >= 3)
    numDarts = atoi(argv[2]);

   MPI_Init(&argc, &argv);
   MPI_Comm_size(MPI_COMM_WORLD, &world_size);

   printf("world size = %i\n", world_size);
   if (world_size != 1)
        printf("Top heavy with management\n");

   int numDartsWorker = numDarts/numWorkers;
   int numDartsMaster = numDarts/numWorkers + (numDarts % numWorkers); //the master computes the leftover
   args[0] = malloc(256 * sizeof(char));
   sprintf(args[0], "%i", numDartsWorker);
   printf("argument passing to workers: %s\n", args[0]);
   /*
    * Now spawn the workers. Note that there is a run-time determination
    * of what type of worker to spawn, and presumably this calculation must
    * be done at run time and cannot be calculated before starting
    * the program. If everything is known when the application is
    * first started, it is generally better to start them all at once
    * in a single MPI_COMM_WORLD.
    */
   printf("About to call MPI_Comm_spawn with %i workers...\n", numWorkers);
   int resultLen = 0;

   double myresult = dboard(numDartsMaster);
   printf("parent result is %.9f\n", myresult);


   //the master counts as a worker, hence the -1
   MPI_Comm_spawn("piworker", args, numWorkers-1, MPI_INFO_NULL, 0, MPI_COMM_SELF,
                   &everyone, MPI_ERRCODES_IGNORE);

   double pisum = 24;
   int rc = MPI_Reduce(&myresult, &pisum, 1, MPI_DOUBLE, MPI_SUM, 0, everyone);

   if (rc != MPI_SUCCESS)
        printf("failure on mpi_reduce\n");

   free(args);
   /*
    * Parallel code here. The communicator "everyone" can be used
    * to communicate with the spawned processes, which have ranks 0,..
    * MPI_UNIVERSE_SIZE-1 in the remote group of the intercommunicator
    * "everyone".
    */

   //receive the results
   int i=1;
   MPI_Status status;
   double avgpi = pisum/(double)numWorkers;
   printf("With %i workers, %i darts, estimated value of pi is: %.9f\n", numWorkers, numDarts, avgpi);

   MPI_Finalize();
   return 0;
}

The code for the worker (spawned) program

int main(int argc, char *argv[])
{
   int size;
   MPI_Comm parent;
   MPI_Init(&argc, &argv);
   MPI_Comm_get_parent(&parent);
   if (parent == MPI_COMM_NULL)
        printf("No parent!");
   int taskid;
   MPI_Comm_remote_size(parent, &size);
   MPI_Comm_rank(MPI_COMM_WORLD,&taskid);
   double pisum = 0;
   int resultLen = 0;
   char parentName[256];
   int numDarts;


   if (size != 1)
   {
        printf("Something's wrong with the parent");
        return 1;
   }
   /*
    * Parallel code here.
    * The manager is represented as the process with rank 0 in (the remote
    * group of) the parent communicator.  If the workers need to communicate
    * among themselves, they can use MPI_COMM_WORLD.
    */
   if(argc >= 2)
        numDarts = atoi(argv[1]);
   else
   {
      printf("Error for: %i, number of darts not specified.\n", taskid);
   }
   printf("numDarts for child: %i\n", numDarts);
   printf("argv[1] = %s\n", argv[1]);
   double myPiSum = dboard(numDarts);
   printf("spawned process got result: %.9f\n", myPiSum);
   printf("Spawned process about to send message back to parent\n");
 //MPI_Send((void *)&myPiSum, 1, MPI_DOUBLE, 0, 1, parent);

   int rc = MPI_Reduce(&myPiSum, &pisum, 1, MPI_DOUBLE, MPI_SUM, 0, parent);
   if(rc != MPI_SUCCESS)
        printf("%d: Problem with mpi_reduce\n");


   printf("Sent message back to parent");
   MPI_Finalize();
   return 0;
}

Hopefully, the cause of this will be more apparent to someone with more experience with this. I’ve been trying all sorts of things, which is why I have so many printf calls.

Answer 1

The problem is that the master process dies because of incorrect usage of free():

char* args[1];
...
args[0] = malloc(256 * sizeof(char));
...
free(args);

You are trying to free non-heap (stack) memory and free(args) triggers an abort in modern glibc versions. The correct invocation should be:

free(args[0]);

Other than that, MPI_Reduce does not work the way you expect it work when called with an intercommunicator. You must change the master code so that it passes MPI_ROOT as the root argument to MPI_Reduce and then you have to manually add the master’s value since it would not be used during the reduction (only values from processes in the remote group are being reduced – see here).