Question

I wrote an automation script to parse an XML file on local, pull some urls from the XML, and then download the files pointed to by the urls and store the data(all jpg’s). Every subsequent iteration progressively takes longer and longer. Eventually, the script just hangs and nothing happens. Oh, the memory usage appears to remain constant. The XML parsing works fine, so please don’t dwell on that too much. If I start the script at a game where the script stops altogether, is saves the files to disk in under a second, because it is the first save the script does. What is causing this? Here’s the code:

    <?php

$xml = simplexml_load_file('./playfin.xml', null, LIBXML_NOCDATA);

echo("Getting urls from the XML...\n");
$data = array();
foreach($xml->game as $game) {
  $smlBox = $game->small_boxshot_image->url;
  $lrgBox = $game->large_boxshot_image->url;

  foreach($game->screenshots->screenshot_image as $image) {
    $scrShots[] = array(
      'smlScr' => (string)$image->small_screenshot_image->url,
      'lrgScr' => (string)$image->large_screenshot_image->url  
    );
  }
  $data[] = array(
    'id' => (int)$game->game_id,
    'smlBox' => (string)$smlBox,
    'lrgBox' => (string)$lrgBox,
    'scrShots' => $scrShots
  );
} 
echo("Done pulling urls from XML...\n");

echo("Storing picture data to disk...\n");
$start = time();
$total = 0;
$now = 0;
for($i = 0; $i < count($data); $i++) {
  $total += $now;
  $now = time() - $start - $total;
  echo("Writing game id: ".$data[$i]['id']. 
    ." to disk at time=".(string)$now."          seconds.\n");
  echo("Memory usage: ".(string)memory_get_usage(true)."\n\n");
  $smlBoxStr = '%s_49_60.jpg';
  $lrgBoxStr = '%s_160_189.jpg';
  $smlScrStr = '%s_115_86_%d.jpg';
  $lrgScrStr = '%s_300_300_%d.jpg';

  writeData($data[$i]['smlBox'], sprintf($smlBoxStr, $data[$i]['id']), 'boxshot');
  writeData($data[$i]['lrgBox'], sprintf($lrgBoxStr, $data[$i]['id']), 'boxshot');
  for($j = 0; $j < count($data[$i]['scrShots']); $j++) {
    writeData(
      $data[$i]['scrShots'][$j]['smlScr'], 
      sprintf($smlScrStr, $data[$i]['id'], $j),
      'screenshot'
    );
    writeData(
      $data[$i]['scrShots'][$j]['lrgScr'], 
      sprintf($lrgScrStr, $data[$i]['id'], $j),
      'screenshot'
    );
  }  
}
echo("Done storing!\n");
echo("Done!\n");

function writeData($url, $file, $type) {
  $headers = get_headers($url, 1);
  $awayPic = fopen($url, 'rb');
  $localPic = fopen("./$type/$file", 'wb');
  while($picData = fread($awayPic, (int)$headers['Content-Length'])) {
    fwrite($localPic, $picData);
  }
  fclose($awayPic);
  fclose($localPic);
}  

?>

Here’s the log file that was produced:

Getting urls from the XML… Done pulling urls from XML… Storing
picture data to disk… Writing game id: 671850 to disk at time=0
seconds. Memory usage: 77856768

Writing game id: 730950 to disk at time=0 seconds. Memory usage:
77856768

Writing game id: 621650 to disk at time=1 seconds. Memory usage:
77856768

Writing game id: 687250 to disk at time=1 seconds. Memory usage:
77856768

Writing game id: 633950 to disk at time=7 seconds. Memory usage:
77856768

Writing game id: 633850 to disk at time=2 seconds. Memory usage:
77856768

Writing game id: 720950 to disk at time=2 seconds. Memory usage:
77856768

Writing game id: 572250 to disk at time=3 seconds. Memory usage:
77856768

Writing game id: 674950 to disk at time=3 seconds. Memory usage:
77856768

Writing game id: 731450 to disk at time=3 seconds. Memory usage:
77856768

Writing game id: 656350 to disk at time=4 seconds. Memory usage:
77856768

Writing game id: 653550 to disk at time=4 seconds. Memory usage:
77856768

Writing game id: 585550 to disk at time=4 seconds. Memory usage:
77856768

Writing game id: 736750 to disk at time=5 seconds. Memory usage:
77856768

Writing game id: 671350 to disk at time=5 seconds. Memory usage:
77856768

Writing game id: 696250 to disk at time=5 seconds. Memory usage:
77856768

Writing game id: 645550 to disk at time=6 seconds. Memory usage:
77856768

Writing game id: 625650 to disk at time=6 seconds. Memory usage:
77856768

Writing game id: 696850 to disk at time=6 seconds. Memory usage:
77856768

Writing game id: 709550 to disk at time=7 seconds. Memory usage:
77856768

Writing game id: 575750 to disk at time=7 seconds. Memory usage:
77856768

Writing game id: 651950 to disk at time=7 seconds. Memory usage:
77856768

Writing game id: 685350 to disk at time=8 seconds. Memory usage:
77856768

Writing game id: 724150 to disk at time=9 seconds. Memory usage:
77856768

Writing game id: 522250 to disk at time=8 seconds. Memory usage:
77856768

Writing game id: 610350 to disk at time=10 seconds. Memory usage:
77856768

Writing game id: 645050 to disk at time=9 seconds. Memory usage:
77856768

Writing game id: 716950 to disk at time=9 seconds. Memory usage:
77856768

Writing game id: 672750 to disk at time=10 seconds. Memory usage:
77856768

Writing game id: 568650 to disk at time=11 seconds. Memory usage:
77856768

Writing game id: 668650 to disk at time=11 seconds. Memory usage:
77856768

Writing game id: 417950 to disk at time=11 seconds. Memory usage:
77856768

Writing game id: 497950 to disk at time=13 seconds. Memory usage:
77856768

Writing game id: 567950 to disk at time=12 seconds. Memory usage:
77856768

Writing game id: 692350 to disk at time=13 seconds. Memory usage:
77856768

Writing game id: 450950 to disk at time=13 seconds. Memory usage:
77856768

Writing game id: 452750 to disk at time=14 seconds. Memory usage:
77856768

Writing game id: 666450 to disk at time=14 seconds. Memory usage:
77856768

Writing game id: 754550 to disk at time=15 seconds. Memory usage:
77856768

Writing game id: 659050 to disk at time=15 seconds. Memory usage:
77856768

Writing game id: 712350 to disk at time=15 seconds. Memory usage:
77856768

Writing game id: 719250 to disk at time=16 seconds. Memory usage:
77856768

Writing game id: 529250 to disk at time=15 seconds. Memory usage:
77856768

Writing game id: 685150 to disk at time=17 seconds. Memory usage:
77856768

Writing game id: 736450 to disk at time=17 seconds. Memory usage:
77856768

Writing game id: 252750 to disk at time=17 seconds. Memory usage:
77856768

Writing game id: 719150 to disk at time=17 seconds. Memory usage:
77856768

Writing game id: 461150 to disk at time=18 seconds. Memory usage:
77856768

Writing game id: 699450 to disk at time=18 seconds. Memory usage:
77856768

Writing game id: 523550 to disk at time=18 seconds. Memory usage:
77856768

Writing game id: 451050 to disk at time=20 seconds. Memory usage:
77856768

Writing game id: 768350 to disk at time=19 seconds. Memory usage:
77856768

Writing game id: 724650 to disk at time=20 seconds. Memory usage:
77856768

Writing game id: 676550 to disk at time=21 seconds. Memory usage:
77856768

Writing game id: 730850 to disk at time=21 seconds. Memory usage:
77856768

Writing game id: 558250 to disk at time=22 seconds. Memory usage:
77856768

Writing game id: 674750 to disk at time=22 seconds. Memory usage:
77856768

Writing game id: 695450 to disk at time=22 seconds. Memory usage:
77856768

Writing game id: 682950 to disk at time=22 seconds. Memory usage:
77856768

Writing game id: 706450 to disk at time=24 seconds. Memory usage:
77856768

Writing game id: 546450 to disk at time=24 seconds. Memory usage:
77856768

Writing game id: 575350 to disk at time=24 seconds. Memory usage:
77856768

Writing game id: 616550 to disk at time=25 seconds. Memory usage:
77856768

Writing game id: 648250 to disk at time=26 seconds. Memory usage:
77856768

Writing game id: 763750 to disk at time=25 seconds. Memory usage:
77856768

Writing game id: 613850 to disk at time=26 seconds. Memory usage:
77856768

Writing game id: 645450 to disk at time=25 seconds. Memory usage:
77856768

Writing game id: 695950 to disk at time=33 seconds. Memory usage:
77856768

Writing game id: 661050 to disk at time=27 seconds. Memory usage:
77856768

Writing game id: 461050 to disk at time=27 seconds. Memory usage:
77856768

Writing game id: 693150 to disk at time=29 seconds. Memory usage:
77856768

Thanks!

Answer 1

The problem is that you’re not clearing $scrShots in the outer loop while parsing the XML. So each iteration not only has the screenshots of the current game in it, but all of the previous ones as well. (This way, you download the images from the first game, then the ones from the first game plus from the second game, then the first plus second plus third and so on, which of course takes longer and longer.)

Try clearing the array before adding the next round to it:

$scrShots = array();
foreach($game->screenshots->screenshot_image as $image) {
    $scrShots[] = array(
    ...