Question

use strict;
use warnings;
use Parallel::ForkManager;

my $log = "/scripts/downloads/test.log";
  print "Check the $log file\n" and open(LOG,">$log");
   *STDERR = *LOG;
  *STDOUT = *LOG;

my $pmm=new Parallel::ForkManager(9);

my @arr=(1..200);
for(@arr){

$pmm->start and next; # do the fork
print $_."\n";
$pmm->finish; # do the exit in the child process
}

$pmm->wait_all_children;

open(READ,"$log") or die $!;
my @check=<READ>;

print "there are ".scalar @arr ." items....";
print "but we only got ".scalar @check." items\n";

This is a simplified version of a script I have going.
In this case, everytime I use more than 9 children I lose anywhere from 3-15 children, sometimes more. The obvious answer is to use less children but in my “real” script if I use less children the script will take many more hours to complete…time we don’t have. Why is it losing children and is there a way to “capture” them and re-run them if they don’t get run?

thx!

Answer 1

You’re having all of your children write to the same logfile, and you haven’t taken any steps to prevent them from overwriting each others’ output. Without being able to reproduce the problem on my machine I can’t say for sure, but I would guess that all N children are actually running, but some of the output is getting clobbered.

To have N processes write to the same file simultaneously without any output getting lost, you have to open the file for append rather than regular write. In C, you would use flags O_WRONLY|O_APPEND with open(2), or mode "a" with fopen(3). The kernel then ensures that all writes go to the very end of the file. (According to the man page, though, this is not reliable over NFS.) You also have to pay attention to how much you write to the file at once, or output from one process might show up in the middle of output from another. I don’t know if either of these things are possible in Perl, but it sounds like you found another solution anyway.