Question

I need to save in real-time to a database all tweets from the Twitter Streaming API, filtering them by out a certain list of words, of course. I’ve achieved it by using tweetstream, defining the list words like this before calling FilterStream():

words = ["word1","two words","anotherWord"]

What I’d like to do, is to be able to add/change/remove any of those values, without stoping the script. To do so, I created a plain text file containing the words I want to be filtered out separated by a line break. Using this code I get the list words just perfectly:

file = open('words.txt','r')
words = file.read().split("\n")

I made those lines work when it starts, but I need it to do it every time it’s going to check the stream. Any ideas?

Answer 1

You could read an updated word list in one thread and process tweets in another one using Queue for communication.

Example:

Thread that reads tweets:

def read_tweets(q):
    words = q.get()
    while True:
        with tweetstream.FilterStream(..track=words,..) as stream:
             for tweet in stream: #NOTE:it requires special handling if it blocks
                 process(tweet)
                 try: words = q.get_nowait() # try to read a new word list
                 except Empty: pass
                 else: break # start new connection

Thread that reads words:

def read_words(q):
    words = None
    while True:
        with open('words.txt') as file:
            newwords = file.read().splitlines()
        if words != newwords:
           q.put(newwords)
           words = newwords
        time.sleep(1)

The main script could look like:

 q = Queue(1)
 t = Thread(target=read_tweets, args=(q,))
 t.daemon = True
 t.start()
 read_words(q)

Instead of polling you could use inotify or similar to monitor changes to the 'words.txt' file.