Question

I have a script that loops through many url based XML files to compare values. There are currently over 50 separate XML files but this number will grow into the hundreds. The compare values are simple and retrieved from a string.

The XML files are well structured and small, 10-26KB, and I only grab 1-2 values from the url based API.

For example:

foreach(); //loop starts, left out details

$value1 = "some_number";

//load all the xml

$request_url = "example.com/file.xml";
$request_url = urlencode($request_url);
$response_xml = @simplexml_load_file($request_url);

$res =  $response_xml;  
$version = $res->version;

if ($value1 > $version)
echo "yay";

endforeach;  //end loop

Now the problem is this becomes slow as beans when getting more that 20 requests, I’d imagine with several hundred it slows to a crawl.

Is there a better method or faster way to do this?

Answer 1

Since fetching the remote files is the slowest part of your iteration, it’s the first place to start to improve performance. I would use cURL to fetch all the files in one swap and after the remote sessions are done, process the data.
cURL provides a curl_multi_exec. php.net/manual/en/function.curl-multi-exec.php which you can feed multiple curl-handles. This will re-use your connection if possible, you can run requests simultaneously.