Question

I’m using Simple HTML DOM to scrape (with permission) some websites. I basically scrape around 50 different websites with statistical data which is updated around four times a day.

As you can imagine it takes times to do the scraping and therefore I need to speed up the process by doing some caching.

My vision is:

DATA-PRESENTATION.php // where all the results are shown

SCRAPING.php // the code that makes the job

I want to set up a cron job on SCRAPING.PHP in a way it executes 4 times a day and save all the data in caché which then will be requested by DATA-PRESENTATION.PHP making the experience for the user way faster.

My question is how can I implement this caché thing? I’m very rookie at PHP, I’ve been reading tutorials but they are not very helpfull and there are just a few so I just couldn’t really learn how to do it.

I know other solution might be implementing a database but I don’t want to do that. Also, I’ve been reading about high end solutions like memcached, but the site is very simple and for personal use, so I don’t need that kind of stuff.

Thanks!!

SCRAPING.PHP

<?php
include("simple_html_dom.php");

// Labour stats
$html7 = file_get_html('http://www.website1.html');
$web_title = $html7->find(".title h1");
$web_figure = $html7->find(".figures h2");

?>

DATA-PRESENTATION.PHP

 <div class="news-pitch">
 <h1>Webiste: <?php echo utf8_encode($web_title[0]->plaintext); ?></h1>
 <p>Unemployment rate: <?php echo utf8_encode($web_figure[0]->plaintext); ?></p>
 </div>

FINAL CODE! Many thanks @jerjer and @PaulD.Waite, I couldn’t really get this done without your help!

Files:

1- DataPresentation.php // here I show the data requested to Cache.html

2- Scraping.php // here I scrape the sites and then save the results to Cache.html

3- Cache.html // here the scraping results are saved

I set up a Cron Job on Scraping.php telling it to overwrite Cache.html each time.

1- DataPresentation.php

<?php
include("simple_html_dom.php");

$html = file_get_html("cache/test.html");
$title = $html->find("h1");
echo $title[0]->plaintext;
?>

2- Scraping.php

<?php
include("simple_html_dom.php");

// by adding "->find("h1")" I speed up things as it only retrieves the information I'll be using and not the whole page.
$filename = "cache/test.html";
$content = file_get_html ('http://www.website.com/')->find("h1");
file_put_contents($filename, $content);
?>

3- Cache.html

<h1>Current unemployment 7,2%</h1>

It loads immediately and by setting things this way I assure there’s always a Caché file to be loaded.

Answer 1

Here is a sample of a file-based caching:

<?php
    // Labour stats
    $filename = "cache/website1.html";
    if(!file_exists($filename)){
        $content = file_get_contents('http://www.website1.html');
        file_put_contents($filename, $content);
    }

    $html7 = file_get_html($filename);
    $web_title = $html7->find(".title h1");
    $web_figure = $html7->find(".figures h2");

?>