Question

I need to manage on-server synchronization between a classic database and a big amount of xls files.

If they are synchronized i perform all the operation over DB and have no problems.

Periodically some of these files are added to the server, and i need to synchronize them. They could be few, but very long, order of hundred thousands rows.

If i catch new of them i start a sync procedure. Sync procedure is proven to last short enought to not exceed Maximmum Execution Time that PHP trig after say 30secs for 1 file. But when i have more files it’s likely to exceed it.

I can not override the PHP time limit.

Right now i have a transaction that backroll if last file sync’tion failed.
And have to reload the page untill i see a sync failed/succeeded message.
This works, but it is not very nice at all.

Choices:

1. with register_shutdown_function print rest of the page ( possible? ) and tell to reload
2. with a custom script same thing in proximity of time limit
3. Any way to complete the sync in background? and maybe notice it to the page through
   ajax maybe?

What would be more the suitable choice? is there any 3rd way?

Answer 1

There is nice trick to bypass the time limit, but it may lag your server if not done properly. Based on this script, I have written pure PHP CRON alternative.
What do you do is, that you make HTTP client disconnect but you keep script running. This is most important – then, the script can call itself without getting stuck on istelf.
To do this, it is impotant to send http header Content-Length.

<?php
ignore_user_abort (true);    //Tells PHP to ignore connection state
ob_start();
echo "Loop started\n";
$size = ob_get_length();
header("Content-Length: $size"); //Send exact output size to make sure client disconnects
header("Connection: close");     //Tells client to disconnect
ob_end_flush();
ob_flush();
flush();     //Clean all buffers
if(session_id()!="")
  session_write_close(); //Close the session so that the user can use other scripts on the site
/*Here, the script runs even if client is connected*/
?>

Now, there is the recursive part which is very dangerous. You should make a file, telling the script that it is already running to avoid multiple threads. Like this:

if(file_exists("secure_key"))
  exit;
else
  file_put_contents("secure_key",time());

Becaus you’ve said that you are syncing some data, you may need one more file (or MySql table) to store information about synchronisation process. This would make it possible to inform user about request progress, maybe using AJAX, like you wanted. This would make whole application look very professional 🙂
You may know that, but if it is FOR or WHILE loop, in which you are syncing, it is also great to save start time in constant so you can quit and save without problems:

define("STARTTIME",time());  //Right now
define("MAXTIME",ini_get('max_execution_time')); //Max execution time
define("SAFE_EXIT_TIME", 4);  //4 seconds to save & quit

/*preparation*/
while(time()-(STARTTIME-SAFE_EXIT_TIME)<MAXTIME) {
    /*Synchronize*/
}
unlink("secure_key");         //delete the antistuck file
get_headers(/*scriopt url*/); //call itself
exit;                         //quit