I have a web app that sends customized emails to my users throughout the day, at times they’ve selected. So at any given time of the day, I might need to do nothing, or send up to ten thousand emails.
Previously I solved this with a single cron job, firing every minute. It would call a script that would find the next thousand emails that needed to be sent. The script would mark them all as “in progress” and then loop over them, trying to send them. The loop is heavy – calling a remote API to send the email, updating the record to “sent”, logging the action. The problem I’ve found is that once in a while, the loop would break, fail, the server would blink, and hundreds of emails would get stuck marked as “in progress”.
I’ve rewritten the loop now to get a single email that needs to be sent, mark it as “in progress”, send it, mark it as “sent”, log it, and then get the next one. I run it for a minute, as many loops as it wants. But this is really cpu intensive, and seems very inefficient, with a lot of db calls.
I’m thinking now that each daily email should have it’s own cron job. Then I end up with ten thousand or more jobs calling a specific url like http://mydomain.com/email/123/send. My concern here is that I will need to update the cron jobs for things like timezones and user preference updates (“send me my email at 6PM instead of 10AM”). So I’m programmatically re-writing my crontab, and possibly often.
How would you solve this problem? As an aside, do you know of any web services designed to solve this problem?
It depends on your exact needs, although 10k cron jobs looks really bad, no matter what.
Maybe you can have a few cronjobs where each picks a batch of mails to send, to achieve parallelism AND avoid delay sending the mails at the bottom ?
Also, for the part of “programatically rewriting my cron”, there are cron libraries available. I’m usually using Quartz for this kind of thing, but it might be too bloated for your needs, so maybe you want to use a lighter library.