Home/ Questions/Q 8702815
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-13T02:44:53+00:00

I have a PHP script, run via cron, that iterates through a list of

  • 0

I have a PHP script, run via cron, that iterates through a list of sites and gathers data for each site using file_get_contents() against an API URL, then stores this data in a MySQL database. This script runs for 60 different site and we’re seeing up to 5MB of serialized PHP data for each URL pull.

The system works swimmingly on my local MAMP server but, as soon as it’s placed on a remote server, it stops behaving reliably. Sometimes it pulls and stores data but stores less records than the local server. It also finishes with a 504 Gateway error. On a different server, it simply doesn’t finish at all. We’ve upped all the time limits in php.ini and compared what I have locally with the remote server and not seeing anything that stands out.

Is there any way to get a PHP script with these multiple repeated actions to run in a more “piecewise” way? Is there any way to “reset the timer” after each site runs? It need to be automated but setting up a cron against each site (3 URLs per site, 60 sites) just sounds like a nightmare to manage.

Thanks in advance.

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    The problem may be caused due to reaching the limit of threads of PHP or the web server, it also may be due to rules in the server about number of connections per unit of time.

    If you can split the load and execute those cron jobs at different moments, you will get better results

    • 0