Question

I’m trying to figure out the most efficient way to running a pretty hefty PHP task thousands of times a day. It needs to make an IMAP connection to Gmail, loop over the emails, save this info to the database and save images locally.

Running this task every so often using a cron isn’t that big of a deal, but I need to run it every minute and I know eventually the crons will start running on top of each other and cause memory issues.

What is the next step up when you need to efficiently run a task multiple times a minute? I’ve been reading about beanstalk & pheanstalk and I’m not entirely sure if that will do what I need. Thoughts???

Answer 1

Either create a locking mechanism so the scripts won’t overlap. This is quite simple as scripts only run every minute, a simple .lock file would suffice:

<?php
  if (file_exists("foo.lock")) exit(0);
  file_put_contents("foo.lock", getmypid());

  do_stuff_here();

  unlink("foo.lock");
?>

This will make sure scripts don’t run in parallel, you just have to make sure the .lock file is deleted when the program exits, so you should have a single point of exit (except for the exit at the beginning).

A good alternative – as Brian Roach suggested – is a dedicated server process that runs all the time and keeps the connection to the IMAP server up. This reduces overhead a lot and is not much harder than writing a normal php script:

<?php
  connect();
  while (is_world_not_invaded_by_aliens())
  {
    get_mails();
    get_images();
    sleep(time_to_next_check());
  }
  disconnect();
?>