Question

I have a perl script which sends a lot of output to multiple subprocesses. I need to be able to close my end of all the pipes and then wait for the subprocesses to finish their work. So far I’ve only succeeded at closing each pipe and waiting for each subprocess to finish one by one.
More concretely, I’m doing something like this:

for ($i=0;$i<24;$i++) {
    my $fh;
    open $fh, "|externalprogram $i";
    $fhs{$i}=$fh;
}

#...now I can write output to the pipes
while (moreworktodo()) {
    $whichone, $data = do_some_work();
    print $fhs{$whichone} $data;
}
#Now I just need to wait for all the subprocesses to finish.  However, they
#need to do a lot of work that can only begin when they've finished reading input.  So I need to close my end of the pipe to indicate I'm finished.
for ($i=0;$i<24;$i++) {
    my $file = $fhs{$i};
    close $file;  #unfortunately, this blocks until process $i finishes
    #meanwhile all the other processes are waiting for EOF 
    #on their STDIN before they can proceed.  So I end up waiting
    #for 24 processes to finish one-at-a-time instead of all at once
}

One way to get all the subprocesses to finish promptly (closing their stdin) is simply to let my script exit without closing the (pipe) filehandles at all, but that’s no good because the script is part of a larger job that needs the subprocess’ work to actually be done before proceeding.

What is a simple way to close each subprocesses’ stdin (so that they can all finish working) and then wait for all of them to finish before proceeding? I’ve tried forking off a child to close each pipe but that doesn’t seem to work — only the parent’s “close” actually closes the stdin of the subprocess and waits for the subprocess to finish.

Answer 1

I would create the pipes myself and not use open(P, "|external-program").
Then you can close the pipe and not wait for the child process to exit.

Example of opening a pipe to a child process yourself:

sub spawn {
  my ($cmd) = @_;

  pipe(my $rp, $wp) or die "pipe failed: $!";

  my $pid = fork();
  die "fork: $!" unless defined($pid);
  if ($pid) {
    # parent
    close($rp);
    return ($wp, $pid);
  } else {
    # child
    close($wp);
    open(STDIN, "<&", $rp);
    exec($cmd) or die "exec: $!";
  }
}

sub main {
  $| = 1;
  my ($wp, $pid) = spawn("./child");
  for (1..10) {
    print {$wp} "sending $_\n";
  }
  close($wp);
  print "done\n";
 }

 main();

Here’s a sample child program to test that close() is NOT waiting for the child to exit:

# file: ./child
while (<STDIN>) {
  print "got: $_";
  sleep(2);
}

The last piece of the puzzle is to asynchronously wait for the child processes to exit.
This can be done with a $SIG{CHLD} handler, or, alternatively, here is a simplistic “join_children” function:

my @child_ids = (1..24); # or whatever ids you want to use
my %pipe;                # hash map from child_id -> pipe handle

sub join_children {
  for my $id (@child_ids) {
    close( $pipe{$id} );
  }

  my $count = scalar(@child_ids);
  while ($count > 0) {
    wait;
    $count--;
  }
}