Id like to run a cron job once a week which runs a php script.
The script will need to get all users from a database and run another script which sends each user an email with a report with data pulled from an external API.
The problem is, there is a 100mb memory limit on every cron job. If there are thousands of users in my db and I need to retrieve lots of data from the external API I will soon exceed the memory limit.
Is there a way to work around this by breaking up the php scripts? What would your strategy be?
If you want to break up the script, you can leave the weekly cron but instead of sending mails you can queue it into a database table. Then, using a second cron that run every 5 or 10 minutes, you can read the database mail queue (searching for max 50 or 100 rows) and if you find something, you send a chunk of emails…
In general, this strategy (huge queue loading, smaller queue processing in chunks) allow you to split execution of large processes.