Question

I have a daily cron job that grabs 10 users at a time and does a bunch of stuff with them. There are 1000’s of users that are part of that process every day. I don’t want the cron jobs to overlap as they are heavy on server load. They call stuff from various APIs so I have timeouts and slow data fetching to contend with.

I’ve tried using a flag to mark a cron job as running, but it’s not reliable enough as some PHP scripts may time out or fail in various ways.

Is there a good way to stop a cron job from calling the PHP script multiple times, or controlling the number of times it is called, so there are only 3 instances for example?

I’d prefer a solution in the PHP if possible.

At the moment I’m storing the flag as a value in a database, is using a lock type file any better as in here https://stackoverflow.com/questions/851872/does-a-cron-job-kill-last-cron-execution ?

Answer 1

I used a dirty way of tracking the number of the scripts being executed via a table in a database. A launched script inserts a row with an id and the field started_time. On exit it removes his row. Too old rows are considered as “failed\dead scripts”.

The worker scripts aren’t launched by cron directly. Cron launches “the launcher script” every 1 second or so, which checks the table for the number of active workers and spawns more workers if needed.

Such a scheme is working online for me for 2.5 years already. Just grabbing some constantly updated content.