Question

I’m struggling trying to figure out how to pull in tweets from the live Twitter stream. I want to keep the stream connection open and render the tweets onto a map as they come in.

I think my problem is in tweetmap.php, as I don’t know how to process the stream as it’s updated. Furthermore, I don’t think I’m checking the position of the last tweet correctly (by the tweet’s ID).

I’ve been using THIS TUTORIAL as my guidance, but I would like to bypass storing them into a database and just process them on the fly.

poll() (in tweetmap.js) is called when the page first loads.

tweetmap.php:

<?php
    $opts = array(
        'http'=>array(
            'method'    =>  "POST",
            'content'   =>  'track=lol',
        )
    );
    $context = stream_context_create($opts);
    while (1){
        $instream = fopen('https://stream.twitter.com/1/statuses/filter.json','r' ,false, $context);
        while(! feof($instream)) {
            if(! ($line = stream_get_line($instream, 20000, "\n"))) {
                continue;
            } else{
                $tweets = json_decode($line);
                echo $tweets;
                flush();
            }
        }
    }
?>

tweetmap.js:

var last = '';
var timeOut;

function getTweets(id){
    $.getJSON("./php/tweetmap.php?start=" + id,
    function(data){
        $.each(data, function(count,item){
            harvest(item);
            last = item.id;
        });
    });
}

function harvest(tweets) {
    for (var i = 0; i < tweets.results.length; i++) {
        if (tweets.results[i].geo !== null) {
            mapTweet(tweets.results[i]);
        }
    }
}

function mapTweet(tweetData) {
    var tipText;
    var coordinates = projection([tweetData.geo.coordinates[1], tweetData.geo.coordinates[0]]);
    [...]
    // Determines the coordinates of the tweet and adds a circle
    addCircle(coordinates, tipText);
}

function addCircle(coordinates, tipText, r) {
    // Adds and SVG circle to the map
    addTipsy(tipText, tweetNumber);
}

// add tipsy tweet-tip 
function addTipsy(tipText, num) {
    // Adds a hover tip of the tweet on the map
    [...]
}

// Draw the map (is also called to redraw when the browser is resized)
function draw(ht) {
    [...]
    // Draws the SVG map
}

function poll(){
    timeOut = setTimeout('poll()', 200); // Calls itself every 200ms
    getTweets(last);
}

$(function() {
    poll();
    draw($("#mapContainer").width()/2.25);

    $(window).resize(function() {
        if(this.resizeTO) clearTimeout(this.resizeTO);
        this.resizeTO = setTimeout(function() {
            $(this).trigger('resizeEnd');
        }, 500);
    });

    $(window).bind('resizeEnd', function() {
        var height = $("#mapContainer").width()/2.25;
        $("#mapContainer svg").css("height", height);
        draw(height);
    });
});

Answer 1

You are trying to treat an HTTP request as a continuous stream while that is not how it works. HTTP is a stateless protocol that handles a request and then closes. You have two options to do what you are trying to do:

1. Use multiple requests, probably on an interval of a second. This will create the appearance of new tweets being realtime. You may have to track the time since last requests so only new tweets are delivered.

2. Use Websocket. This will provide a communication channel like what you are expecting. However, its not supported on all browsers.