Question

I have a Perl script which forks a number of sub-processes. I’d like to have some kind of functionality like xargs --max-procs=4 --max-args=1 or make -j 4, where Perl will keep a given number of processes running until it runs out of work.

It’s easy to say fork four process and wait for them all to complete, and then fork another four, but I’d like to keep four or n processes running at the same time, forking a new process as soon as one completes.

Is there a simple way in Perl to implement such a process pool?

Answer 1

Forks::Super can handle this requirement.

use Forks::Super MAX_PROC => 5, ON_BUSY => [ block | queue ];

Calls to fork() can block until the number of active subprocesses falls below 5, or you can pass additional parameters to the fork call and the tasks to perform can queue up:

fork { sub => sub { ... task to run in subprocess ... } }

When one subprocess finishes, another job on the queue will start up.

(I am the author of this module).