Question

I am currently using CURL via a php script running as daily cron to export product data in csv format from a site’s admin area.

The normal way of exporting data will be to go to the Export page in a browser, and set the configuration, then click on “export data” button. But as the number of products I am exporting is very large, and it takes more than 5-10 mins to export the data, I’ve decided to use php’s curl function to mimic this on a daily basis via cron.

Previously, it is working fine, but recently as I increased the number of products in the store by 500+, the script fails to return the exported data. Testing it manually via clicking on the “export” button in a browser, does return the data correctly. Thus there is no “timeout” issue with running the export in a browser manually.

I’ve tested and by removing/decreasing the number of products (thus the time needed), the php-curl script works fine again when run from cron.

So I suspect that it has something to do with timeouts issue, specifically with the curl function in php.

I’ve set both CURLOPT_TIMEOUT and CURLOPT_CONNECTTIMEOUT to ‘0’ respectively to try.
In the php-curl script, I’ve also set “set_time_limit(3000)”. But still it does not work, and the request will timeout, with the script failing to return with a complete set of csv data.

Any help in helping me resolve/understand this issue will be much appreciated!

Edited: Added part of the code where the curl script is calling the export function.

$interface = new StoreInterface(); 
echo "Start exporting at " .  date('l jS \of F Y h:i:s A') . "\n";
set_time_limit(3000);
$result_html = $interface->exportProducts();
//parse $result_html to only retain the csv format
preg_match('/<pre>(.*)<\/pre>/s',$result_html[0],$output);
if(strlen($output[1])<10) {   //debugging for now
echo "Export did not happen correctly. Quit\n";
    die('Export unsuccessful');
}
file_put_contents($output_path,$output[1]);
echo "Script completed. Thank you! \n";

Answer 1

You can use the following curl options to log all the curl transfer details to a log file and check for any issues.

$fp = fopen('./debug/transfer.log');
curl_setopt($curl, CURLOPT_VERBOSE, TRUE);
curl_setopt($curl, CURLOPT_STDERR, $fp);

Also I believe that the CURLOPT_TIMEOUT option doesn’t support specifying a “0” value for an indefinite timeout. You need to specify the maximum timeout value here.