Question

I’m building a small application that requires people to upload images by email. It is built in PHP (no framework) with MySQL and S3.

So far, in my scenario: emails are stored on a POP3 account. A script runs every minute, fetches the oldest email, resizes the image, uploads it to S3, store its path in the DB, deletes the email.

In a larger scale, how would this be managed? Is the cron job the best way to handle this type of situation? What if the process takes more than a minute: it will overlap and eventually fail, right? Or what if it takes less than a minute? I’d get unwanted idle time considering I would have more than 60 requests an hour in a bigger scale…

Perhaps I should use a .forward file to process emails, but again I would not control the flow.

I believe most of these scenarios work, I’m just curious regarding best practices.

Thanks!

Answer 1

A slightly modified approach could be:

• Run your script from the cron every minute
• Your script check if another instance of it is already running and if it is the case just exists
• The running script process the remaining queue until is empty or a number of elements (ex. max 10)

I once had a backup procedure that was backing up customer directories every hour, if the last completed backup was older than a certain amount of time. This worked great until some customer had too many Gb of data and the script was taking more than a hour to do the backup.

Without the check, the next hour the script is going to run again the same customer, and that would take also more than a hour, and so on, until the machine becomes unresponsive with a very high runlevel.

The fix that I’ve implemented was the check described, if another instance was running, just exit and wait for the next cycle. After that fix I never had a problem for years.