Question

I am using Python and Requests to request data from Twitter’s streaming API. I would like to request data with a set of parameters, and after a period of time, change the request with a new set of parameters.

In the following simple, working example, I ask the Twitter streaming API for tweets with the keyword ‘python.’ After an hour, I ask the API for the keyword ‘ruby.’ However, to do this, I am creating a new requests object; I am not changing the original object.

import requests
import json
import time

USER = 'user'
PW = 'pw'

def make_request(keyword):
    r = requests.post('https://stream.twitter.com/1/statuses/filter.json',
    data={'track': keyword}, auth=(USER,PW))

    for line in r.iter_lines():
        if line:
            print json.loads(line)

        if time.time() > start_of_last_request + 3600:
            break

count = 0

keywords = ['python', 'ruby']

while count < 2:
    start_of_last_request = time.time()

    make_request(keywords[count])

    count = count + 1

Ultimately, I will need to create a new query every hour and I am worried that I will be creating too many connections.

My questions are: Is there a better way to change the request to the Twitter API? Am I actually creating multiple connections? If so, how do I avoid this? (There’s a suggestion on how to close the previous connections, but the solution doesn’t quite make sense to me.)

I would appreciate any help. Thank you!

Answer 1

As you linked, in order to close the connection after you are done with the request, set the keep-alive value to False instead of the default True:

r = requests.post('https://stream.twitter.com/1/statuses/filter.json',
               data={'track': keyword}, auth=(USER,PW),
               config={'keep_alive':False})

This will make sure that the connection is closed once you’re done with it.