Question

I am getting at extremely fast rate, tweets from a long-lived connection to the Twitter API Streaming Server. I proceed by doing some heavy text processing and save the tweets in my database.

I am using PyCurl for the connection and callback function that care of text processing and saving in the db. See below my approach who is not working properly.

I am not familiar with network programming, so would like to know:
How can use Threads, Queue or Twisted frameworks to solve this problem ?

def process_tweet():
    # do some heaving text processing


def open_stream_connection():
    connect = pycurl.Curl()
    connect.setopt(pycurl.URL, STREAMURL)
    connect.setopt(pycurl.WRITEFUNCTION, process_tweet)
    connect.setopt(pycurl.USERPWD, "%s:%s" % (TWITTER_USER, TWITTER_PASS))
    connect.perform()

Answer 1

Here’s simple setup if you are OK with using a single machine.

1 thread accepts connections. After a connection is accepted, it passes the accepted connection to another thread for processing.

You can, of course, use processes (e.g, using multiprocessing) instead of threads, but I’m not familiar with multiprocessing to give advice. The setup would be the same: 1 process accepts connections, then passes them to subprocesses.

If you need to shard the processing across multiple machines, then the simple thing to do would be to stuff the message into the database, then notify the workers about the new record (this will require some sort of coordination/locking between the workers). If you want to avoid hitting the database, then you’ll have to pipe messages from your network process to the workers (and I’m not well versed enough in low level networking to tell you how to do that :))