Question

I have a program that is being symlinked to multiple directories e.g.

/main/foo.pl
/run1/foo.pl -> /main/foo.pl
/run2/foo.pl -> /main/foo.pl
/run3/foo.pl -> /main/foo.pl
/run4/foo.pl -> /main/foo.pl

They’re being run as cron jobs, hence I have the following entries in the crontab:

*/2 * * * * /run1/foo.pl
*/2 * * * * /run2/foo.pl
*/2 * * * * /run3/foo.pl
*/2 * * * * /run4/foo.pl

A snippet of foo.pl is as below:

use Fcntl qw(:flock);
use autodie qw(:all);
open my $self, '>', "$FindBin::Bin/lockme";
flock( $self, LOCK_EX|LOCK_NB )
    or die "Cannot acquire lock, already running!";

{
    my $long_proc = Process->new();
    $long_proc->run();
}        

You get the idea, each of the cron process can only run once because there is a lock semaphore check. But run1, run2, run3, and run4 can run simultaneously.

Now what I need is that I want to limit the number of process to maximum of four. If someone adds another cron processes like:

New symlinks:

/run5/foo.pl -> /main/foo.pl
/run6/foo.pl -> /main/foo.pl

Additional crontab:

*/5 * * * * /run5/foo.pl
* * * * * /run6/foo.pl

Both run5 and run6 need to be queued whenever run1, run2, run3, and run4 are all still running. Thus at any given time there will be only 4 processes run.

How can I achieve that? Is there any CPAN module that handles it?

Thanks!

Answer 1

Please see comment from @tsee in @pilcrow answer above. Steffen Muller’s IPC::ConcurrencyLimit can control the number of concurrenct processes on cooperative multi processing nicely.