Question

I need every day update ~ 10.000 items in my MySql. I have upload a CSV file and try to update my database data. I am forking with two field. My problem is that works but in some time I get 504 Gateway Time-out and my upload is not able to finish the process.

here is my code

if(is_uploaded_file($_FILES["filename"]["tmp_name"])){
        //      
        move_uploaded_file($_FILES["filename"]["tmp_name"], "".$_SERVER["DOCUMENT_ROOT"]."/upload/".$_FILES["filename"]["name"]);
    $file_path="".$_SERVER["DOCUMENT_ROOT"]."/upload/".$_FILES["filename"]["name"]."";

    $file=file_get_contents("".$file_path."");
    $file=iconv("windows-1251", "utf-8",$file);
    file_put_contents("".$file_path."",$file);

if(!setlocale(LC_ALL, 'ru_RU.utf8')) setlocale(LC_ALL, 'en_US.utf8'); if(setlocale(LC_ALL, 0) == 'C') die('       (ru_RU.utf8, en_US.utf8)');
    if (($handle_f = fopen($file_path, "r")) !== FALSE) { 
        //   csv
        while (($data_f = fgetcsv($handle_f,99999,";"))!== FALSE) {
            //   ean13 
            $sql="SELECT id_product FROM ps_product WHERE reference = '".$data_f[0]."'";
            $id_product = Db::getInstance()->getValue($sql,0);  

            //   ,    
            if ($id_product) {      
                $sql=mysql_query("UPDATE `ps_product` SET `quantity` ='".$data_f[1]."' WHERE `reference`='".$data_f[0]."'");
                echo "<p style='color:green'>Items<b>".$data_f[0]."</b> updated</p>";
            } else{
                echo "<p style='color:red  '>Items<b>".$data_f[0]."</b> not found</p>";
            }
        }
      echo "<b>Update is complited</b>";  

    }else{
        echo "Can`t open imported file";
    }
}else{
    echo '
    <h2>Quantity update:</h2>
    <form action="'.$_SERVER["PHP_SELF"].'" method="post" enctype="multipart/form-data">
              <input type="file" name="filename"><br>
              <input type="submit" value="Load"><br>
    </form>
    ';
}

Now I split in excel for many files with ~ 1000 per fail and update database. It takes some time. Can you give me an idea, or maybe I have some error in my code? Update using ajax as I think will not help.

Answer 1

504 Bad Gateway Timeout is when your gateway/proxy server did not get a timely response from your processing server. In such cases, even if the processing server does finish all the updates, since it did not send any messages to your proxy to let it know that processing was still going on, the proxy closes the connection.

Using ajax for updates is not a very good idea in this case. However there is an ajax design pattern called the heartbeat pattern which can help you out. The basic idea is this:

1. You make a normal call to the processing page to start the CSV update
2. Additionally you set a javascript timer in your page which makes an ajax call to a server page every X minutes
3. The server responds with a simple message. Can be just a single character even
4. Your client side ajax can ignore this message, or print it or whatever
5. After X minutes the same process happens.
6. In the meantime the CSV processing you initiated in step one is still continuing

This continuous message request and response is called a heartbeat. It helps in keeping the connection alive while your server processing takes place. Once the server processing is over and you get a confirmation, you can kill the timer and close the connection as needed.

Of course on the processing server side, you can do additional things like increase timeout lenght, add database indexes etc to make sure that a php process timeout does not take place.

Note however that the process timeout is different from a Gateway timeout.