Question

I need to come up with a way to make a large task faster to beat the timeout.

I have very limited access to the server due to the restrictions of the hosting company.

I have a system set up where a cron visits a PHP file that grabs a csv that contains data on some products. The csv does not contain all of the fields that the product would have. Just a handful of essential ones.

I’ve read a fair number of articles on timeouts and handling csv’s and currently (in an attempt to shave time) I have made a table (let’s call it csv_data) to hold the csv data. I have a script that truncates the csv_data table then inserts data from the csv so each night the latest recordset from the csv is in that table (the csv file gets updated nightly). So far, no timeout problems..the task only takes about 4-5 seconds.

The timeouts occur when I have to sift through the data to make updates to the products table. The steps that it is running right now is like this

1. Get the sku from csv_data table (that holds thousands of records)
2. Select * from Products where products.sku = csv.sku (products table also holds thousands of records to loop through)
3. Get numrows.   
    If numrows<0{no record in products, so skip}. 
    If numrows>1{duplicate entries, don't change anything, but later on report the sku}
    If numrows==1{Update selected fields in the products table with csv data}
4. Go to the next record in csv_data all over again

(I figured outlining the process is shorter and easier than dropping in the code.)
I looked into MySQl views and stored procedures but I am not skilled enough in it to know if it will handle the ‘if’ statement portion.

Is there anything I can do to make this faster to avoid the timeouts?

edit:

I should mention that set_time_limit(0); isn’t doing it. And if it helps, the server uses IIS7 and fastcgi
Thanks for your help.

Update after using suggestions from Jakob and Shawn:

I’m doing something wrong. The speed is definitely faster and the csv sku is incrementing,

but when I tried to implement Shawn’s solution; the query is giving me a PHP Warning: mysql_result() expects parameter 1 to be resource, boolean error.

Can you help me spot what I am doing wrong?

Here is the section of code:

$csvdata="SELECT * FROM csv_update";
    $csvdata_result=mysql_query($csvdata); 
    mysql_query($csvdata);
    $csvdata_num = mysql_num_rows($csvdata_result);
    $i=0;       
while($i<$csvdata_num){

$csv_code=@mysql_result($csvdata_result,$i,"skucode");

$datacheck=NULL;    
$datacheck=substr($csv_code,0,1);

if($datacheck>='0' && $datacheck<='9'){

$csv_price=@mysql_result($csvdata_result,$i,"price");
$csv_retail=@mysql_result($csvdata_result,$i,"retail");
$csv_stock=@mysql_result($csvdata_result,$i,"stock");
$csv_weight=@mysql_result($csvdata_result,$i,"weight");
$csv_manufacturer=@mysql_result($csvdata_result,$i,"manufacturer");
$csv_misc1=@mysql_result($csvdata_result,$i,"misc1");
$csv_misc2=@mysql_result($csvdata_result,$i,"misc2");
$csv_selectlist=@mysql_result($csvdata_result,$i,"selectlist");
$csv_level5=@mysql_result($csvdata_result,$i,"level5");
$csv_frontpage=@mysql_result($csvdata_result,$i,"frontpage");
$csv_level3=@mysql_result($csvdata_result,$i,"level3");
$csv_minquantity=@mysql_result($csvdata_result,$i,"minquantity");
$csv_quantity1=@mysql_result($csvdata_result,$i,"quantity1");
$csv_discount1=@mysql_result($csvdata_result,$i,"discount1");
$csv_quantity2=@mysql_result($csvdata_result,$i,"quantity2");
$csv_discount2=@mysql_result($csvdata_result,$i,"discount2");
$csv_quantity3=@mysql_result($csvdata_result,$i,"quantity3");
$csv_discount3=@mysql_result($csvdata_result,$i,"discount3");

    $count_check="SELECT COUNT(*) AS totalCount FROM products WHERE skucode = '$csv_code'";
    $count_result=mysql_query($count_check); 
    mysql_query($count_check);
    $totalCount=@mysql_result($count_result,0,'totalCount');
    $loopCount = ceil($totalCount / 25);
    for($j = 0; $j < $loopCount; $j++){

    $prod_check="SELECT skucode FROM products WHERE skucode = '$csv_code' LIMIT ($loopCount*25), 25;";
    $prodresult=mysql_query($prod_check); 
    mysql_query($prod_check);
    $prodnum =@mysql_num_rows($prodresult);
    $prod_id=@mysql_result($prodresult,0,"catalogid");


    if($prodnum<1){ 
    echo "NOT FOUND:$csv_code<br>";
    $count_sku_not_found=$count_sku_not_found+1;
    $list_sku_not_found=$list_sku_not_found." $csv_code";}

    if($prodnum>1){ 
    echo "DUPLICATE:$csv_ccode<br>";    
    $count_duplicate_skus=$count_duplicate_skus+1;
    $list_duplicate_skus=$list_duplicate_skus." $csv_code";}


if ($prodnum==1){
///This prevents an overwrite from happening if the csv file doesn't produce properly
    if ($csv_price!="" OR $csv_price!=NULL)
    {$sql_price='price="'.$csv_price.'"';}

    if ($csv_retail!="" OR $csv_retail!=NULL)
    {$sql_retail=',retail="'.$csv_retail.'"';}

    if ($csv_stock!="" OR $csv_stock!=NULL)
    {$sql_stock=',stock="'.$csv_stock.'"';}

    if ($csv_weight!="" OR $csv_weight!=NULL)
    {$sql_weight=',weight="'.$csv_weight.'"';}

    if ($csv_manufacturer!="" OR $csv_manufacturer!=NULL)
    {$sql_manufacturer=',manufacturer="'.$csv_manufacturer.'"';}

    if ($csv_misc1!="" OR $csv_misc1!=NULL)
    {$sql_misc1=',misc1="'.$csv_misc1.'"';}

    if ($csv_misc2!="" OR $csv_misc2!=NULL)
    {$sql_pother2=',pother2="'.$csv_misc2.'"';}

    if ($csv_selectlist!="" OR $csv_selectlist!=NULL)
    {$sql_selectlist=',selectlist="'.$csv_selectlist.'"';}

    if ($csv_level5!="" OR $csv_level5!=NULL)
    {$sql_level5=',level5="'.$csv_level5.'"';}

    if ($csv_frontpage!="" OR $csv_frontpage!=NULL)
    {$sql_frontpage=',frontpage="'.$csv_frontpage.'"';}



$import="UPDATE products SET $sql_price $sql_retail $sql_stock $sql_weight $sql_manufacturer $sql_misc1 $sql_misc2 $sql_selectlist $sql_level5 $sql_frontpage    $sql_in_stock WHERE skucode='$csv_code'";
 mysql_query($import) or die(mysql_error("error updating in products table"));


echo "Update ".$csv_code." successful ($i)<br>";

$count_success_update_skus=$count_success_update_skus+1;
$list_success_update_skus=$list_success_update_skus." $csv_code";



//empty out variables 
$sql_price='';
$sql_retail='';
$sql_stock='';
$sql_weight='';
$sql_manufacturer='';
$sql_misc1='';
$sql_misc2='';
$sql_selectlist='';
$sql_level5='';
$sql_frontpage='';
$sql_in_stock='';
$prodnum=0;

}
}
$i++;
}

Answer 1

Is it timing out before the first row is returned or is it between rows during the read? One good practice bit would be to handle your query in chunks; do a count first to see how many records you are dealing with for the SKU, the loop through smaller chunks (the size of these chunks would depend on how many things you have to do with each row). Your updated workflow would look more like this:

1. Get next SKU from CSV
2. Get a total count: SELECT COUNT(*) AS totalCount FROM products WHERE products.sku = csv.sku
3. Determine chunk size (using 25 for this demo)
4. loopCount = ceil(totalCount / 25)
5. Loop through all results using a loop like this: for($i = 0; $i < loopCount; $i++)
6. Inside your loop you should be running a query like this: SELECT * FROM products WHERE products.sku = csv.sku LIMIT (loopCount*25), 25

You will want to use a constant order for your SELECT chunks; your unique ID would probably be best.