Question

I have a twitter feed on my website which is powerd by the jquery tweet plugin tweet.seaofclouds.com and some other json plugins that take a json feed from a 3rd party website.

The problem is dat the twitter api only allows 150 calls per hour, so with 40 visitors an hour with an average of 5 pageviews per visitor I go way past that max. Especially since twitter disabled caching on the feed.

Also, there is the coockie law problem. Twitter drops cookies when the feed is requested and I don’t want them because they need to have permission to be dropped, so I want to disable them all the way.

Also my website is SSL secure and I want to load as minimal external resources as possible, I want it all localised.

How do I locally cache these json feeds?

Answer 1

For this problem I have written my own database storing mechanism to store the json feeds and fetch them when needed and return them. That way I only need to fetch every 5 minutes and the amount of visitors/pageviews I get is irrelevant.

Here is the database creation code in mysql

CREATE TABLE IF NOT EXISTS `twitterbackup` (
  `url` text NOT NULL,
  `tijd` int(11) NOT NULL,
  `inhoud` text NOT NULL,
  FULLTEXT KEY `url` (`url`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;

Then in PHP I have this code with some security checks since you’ll never know what you’ll get

<?php
/* JSON Backup script written by Michael Dibbets
 * Copyright 2012 by Michael Dibbets
 * http://www.facebook.com/michael.dibbets - mdibbets[at]outlook.com
 * Licenced under the MIT license http://opensource.org/licenses/MIT
 */
// Basic sql injection protection.
// Using explode because str_replace fails to many times when certain character combinations exist. 
// Replace, remove as you see fit. This setup works for my server, and str_replace works on another. 
// Use whatever has your fancy
function protect($s)
    {
    $s = mysql_real_escape_string($s);
    $s = implode(" ",explode(";",$s));
    $s = implode(" ",explode("UNION",$s));
    $s = implode(" ",explode("BENCHMARK",$s));
    $s = implode(" ",explode("WAITFOR DELAY",$s));
    $s = implode(" ",explode("LOAD_FILE",$s));
    $s = implode(" ",explode("OUTFILE",$s));
    $s = implode(" ",explode("INFORMATION_SCHEMA",$s));
    $s = implode(" ",explode("Char(",$s));
    $s = implode(" ",explode("CAST(",$s));
    return $s;
    }
function get_data($url)
    {
    // Initialise data to have at least something to work with
    $data = "";
        // What time is it?
        $now = strtotime("now");
        // Connect to our database
        $db = mysqli_connect("localhost", "USERNAME", "PASSWORD", "DATABASE");
        if (mysqli_connect_errno($mysqli)) 
            {
            die("ARGH!");
            }
        // Basic protection agains sql injection by banning unsafe words
        $saveurl = protect($url);
        // Count how many times the url has been found.
        $count = $db->query("SELECT count(*) as counter FROM twitterbackup WHERE `url`='$saveurl'")->fetch_assoc();
        // Has the url been found?
        if($count['counter'] == 0)
            {
            // Fetch twitter json 
            $ch = curl_init();
            $timeout = 5;
            curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
            curl_setopt($ch,CURLOPT_URL,$url);
            curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);
            curl_setopt($ch,CURLOPT_CONNECTTIMEOUT,$timeout);
            $data = curl_exec($ch);
            curl_close($ch);
            // make the json data database safe
            $data = str_replace('\\','\\\\',$data);
            $data = mysql_real_escape_string($data);
            //$data = mysql_real_escape_string($data);
            // Enter json data in the database
            $db->query("INSERT INTO `DATABASE`.`twitterbackup` (`url`, `tijd`, `inhoud`) VALUES ('$saveurl', '$now', '$data')");
            // End of we have not found the url
            }
        // If the URL has been found
        else
            {
            // get the values in the database that are connected to the url
            $res = $db->query("SELECT * FROM twitterbackup WHERE `url`='$saveurl'")->fetch_assoc();
            // Is the current json in database younger than five minutes?
            if((int)$res['tijd'] < (int)strtotime("-5 minutes"))
                {
                // Fetch twitter json with curl
                $ch = curl_init();
                $timeout = 5;
                curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
                curl_setopt($ch,CURLOPT_URL,$url);
                curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);
                curl_setopt($ch,CURLOPT_CONNECTTIMEOUT,$timeout);
                $data = curl_exec($ch);
                curl_close($ch);
                // Make the json data safe for the database
                $data = str_replace('\\','\\\\',$data);
                $data = mysql_real_escape_string($data);
                // Update the database with the most recent feed
                $db->query("UPDATE  `DATABASE`.`twitterbackup` SET 
                            `tijd` =  '$now',
                            `inhoud` =  '$data' 
                            WHERE  `twitterbackup`.`url` =  '$saveurl'");
                // End if the url has been found and lifetime is older than five minutes
                }
            // If the lifetime isn't older then 5 minutes
            else
                {
                // return database content
                $data = $res['inhoud'];
                }
            // end of if we have found the url
            }
          // try to beat mysql_real_escape_string to return valid json. Always check valid json returend and edit this if it fails at some piece. Try http://jsonlint.com/
          $data = str_replace('\\"','"',$data);
          // implode because str_replace won't do everthing for some reason I can't understand.
          $data = implode('\\',explode('\\\\',$data));
          // Data retourneren
          return $data;
        // end check if it's from the twitter api
        }
// End of function get_data();
// How long may the json be cached by browser(to stop unneccesary requests in this case 5 minutes)
$seconds_to_cache = 5*60;
$ts = gmdate("D, d M Y H:i:s", time() + $seconds_to_cache) . " GMT";
header("Expires: $ts");
header("Pragma: cache");
header("Cache-Control: max-age=$seconds_to_cache");
header('Content-type: application/json');
echo get_data($_GET['url']);
?>

Then in the twitter.js you only need to replace the getJSON url to point to your local server as follows(somewhere at the bottom of jquery.tweet.js you’ll find this line)

Find: $.getJSON(build_api_url()).success(function(data){

Replace:

// For debug purposes
// console.log("/scripts/twitter/tweet/curlthispage.php?url="+encodeURIComponent(build_api_url()));
        $.getJSON("/scripts/twitter/tweet/curlthispage.php?url="+encodeURIComponent(build_api_url())).success(function(data){