Question

here I have a some script but 3 foreach function is too long and I have and INterneal server error. Can I fix that with curl or something like that.

Here is the code:

<?php

$dom = new DOMDocument();
@$dom->loadHTMLFile('http://www.auto-types.com');
$xpath = new DOMXPath($dom);
$entries = $xpath->query("//li[@class='clearfix_center']/a/@href");
$output = array();
foreach($entries as $e) {  
  $dom2 = new DOMDocument();
  @$dom2->loadHTMLFile('http://www.auto-types.com' . $e->textContent);
  $xpath2 = new DOMXPath($dom2);
  $data = array();
  $items = $xpath2->query("//div[@class='modelImage']/a/@href");
  $links = array();
  foreach($items as $item) {
    $dom3 = new DOMDocument();
    @$dom3->loadHTMLFile('http://www.auto-types.com' . $item->textContent);
    $xpath3 = new DOMXpath($dom3);   
    $konacno = array();
    $krajs = $xpath3->query("//div/@onclick");
        foreach ($krajs as $kraj) {
            $konacno[] = $kraj->textContent;
        }
  }

  $data['newLinks'] = implode(', ', $konacno);

  $output[] = $data;
}

echo '<pre>' . print_r($output, true) . '</pre>';

?>

Answer 1

You should better do this job using some kind of persistant queue. This could be a database table or even a textfile (remember locking here).

Whenever you need to request a new page, put request into queue. If you found any data, store it. When you’re done, fetch the next job from the queue. To be sure not to exceed script execution time limit, you can do an in-browser forward (if you haven’t got another chance to run your job longer).

You’d even be able to run multiple workers in parallel scraping that page!