Question

Background:
I haven’t had much experience with multi-process Perl scripts. I have a data cleanup process for the FooService that is taking over 12 hours to complete, and when I investigated, I found that almost all of that time was spent waiting for the FooClient to return me data. I was looking into a multi-process way to do the task, and a coworker recommended Parallel::Fork::BossWorkerAsync over the simple fork() I was doing before. I liked it since it lowered my memory use by a ton.

Problem:
BossWorkerAsync looks pretty neat, the perldoc is great, and running it in no-write test mode works really well, pushing my execution time under an hour. My only problem is that the documentation doesn’t really explain how shared data works with the “init_handler => &x” construction setting. I want each worker to have its own FooClient, just to avoid any sort of synchronization issues. I went with what I thought was correct, but I’m sort of paranoid about it, and also want to make sure that I’m dealing with this in the most correct way.

Code:

# The number of children to spawn, modify after performance testing
Readonly my $CHILDREN => 40;

# Each child will set their own client
my $client;

my $bw = Parallel::Fork::BossWorkerAsync->new(
    work_handler => \&process_keys,
    init_handler => \&setup_client,
    worker_count => $CHILDREN,
);

send_work($bw);

while ($bw->pending()) {
    my $ref = $bw->get_result();
    # Do stuff with the result
}

$bw->shut_down();
exit;

sub setup_client {
    $client = FooClient->new();
}

Am I handling the $client that I don’t want shared correctly? I kept the same sort of deal I had with my fork() version, where I set the $client after the fork(), but I’m just worried that it’s not the right way to do this.

Answer 1

Yes, you’re using the module, and the init_handler, correctly. The handler is called just after the fork, in each child, before it enters the blocking select loop, waiting for a job.

I’m the author of the module. I’m sorry it took me so long to notice this, and respond. Glad to see the code is being used.

Cheers,
-joe