Question

I have this big function (1300+ lines of code) that takes data from the web and insert it into a local database. Each time the function runs its takes something like 20 seconds to complete and I need to run this function like a million times, so I use set_time_limit(0) to set the PHP time limit to infinite and I loop the function a million times, like this:

for ($ID= '01'; $ID < '999999'; $ID++) {
    getDataFromWeb($conn, $ID);
}

So whats the problem? The problem is that there are a million things that can go wrong and it always does go wrong, and suddenly the code gets stuck in ID 23465 for example, and it just stop getting data but I don’t get any kind of error, its like the loop continues but without inserting anything to database, and because of the ‘no time limit’ I set to PHP then it never stops.

I want to know how I can detect this kind of problem, stop all and show alert. If a I set the time before the function starts and then check it when the function ends, like this:

for ($ID= '01'; $ID < '999999'; $ID++) {
    $time_start = microtime();
    getDataFromWeb($conn, $ID);
    $time_end = microtime();
    if ($time_alert - //... somehow check how time does it takes and stop if its taking too much
}

It will not work because if the function never completes then $time_end will never be set and so on…

So, help please?

Answer 1

Side note: The supplied code will not loop 1,000,000 times. The following will:

for( $id=1 ; $id<=1000000 ; $id++ ) {
    getDataFromWeb( $conn , $id );
}

Also, with regards to your need to have this script run constantly to load content into a database, I would suggest the following:

• I presume that you are using an SQL Table to hold the URLs to be crawled,
• Add a field with a timestamp called ‘loadAttempted’,
• Limit the PHP Script to try and perform the action to maybe 5 times,
• Record the Time the Script attempt to crawl the URL into the ‘loadAttempted’ field,
• Have each loop of the Script perform a search for any URLs where ‘loadAttempted’ is empty, or where it is greater than X minutes ago,
• Add a CRON Job to trigger the Script

This would mean that, up to every minute, the script will be triggered and will try and load 5 URLs. If a URL takes an abnormally long period of time to load (which would mean that the script timed out whilst trying to crawl it) it will cycle back around and be tried again.

You could also use this, or variants on the idea, to get stats for pages which are slower than the rest and/or the average loadtime for the URLs.

Also, if you are wanting to have this running constantly, I would suggest that limiting the PHP script to try and run the getDataFromWeb() function a smaller number of times (like 5)