Question

I’m using Sinatra, EventMachine, DataMapper, SQLite3 and the Twitter Stream API to capture and save tweets. When I run the application from my command line, it seems to continually fail at tweet 50. If I’m not saving the tweets, it can run seemingly forever.

Below is the app code to capture tweets with ‘oscar’ in them, which provided a very quick stream. Just enter your twitter username and password and run at the command line.

require 'rubygems'
require 'sinatra'
require 'em-http'
require 'json'
require 'dm-core'
require 'dm-migrations'

USERNAME = '<your twitter username>'
PASSWORD = '<your secret password>'
STREAMING_URL = 'http://stream.twitter.com/1/statuses/filter.json'

DataMapper.setup(:default, ENV['DATABASE_URL'] || "sqlite3://#{Dir.pwd}/db/development.db")

class Tweet
    include DataMapper::Resource

    property :id,           Serial
    property :tweet_id,     String
    property :username,     String
    property :avatar_url,   String
    property :text,         Text
end

DataMapper.auto_upgrade!

get '/' do
    @tweets = Tweet.all

    erb :index
end

def rip_tweet(line)
    @count += 1

    tweet = Tweet.new   :tweet_id => line['id'],
                :username => line['user']['screen_name'],
                :avatar_url => line['user']['profile_image_url'],
                :text => line['text']

    if tweet.save
        puts @count
    else
        puts "F"
    end
end

EM.schedule do
    @count = 0
    http = EM::HttpRequest.new(STREAMING_URL).get({
        :head => {
            'Authorization' => [ USERNAME, PASSWORD]
        },
        :query => {
            'track' => 'oscars'
        }
    })

    buffer = ""

    http.stream do |chunk|
        buffer += chunk

        while line = buffer.slice!(/.+\r?\n/)
            rip_tweet JSON.parse(line)
        end
    end
end

helpers do
    alias_method :h, :escape_html
end

Answer 1

I’m not sure you can safely mix EM and Sinatra in the same process. You might want to try splitting the Sinatra viewer and the EventMachine downloader into separate programs and processes.