Question

I am learning Python, and I want to start using Twitter’s streaming API. I know how to build a very basic application that reads the contents of a page using urllib2 and have done this successfully in the past. However, the secure connection https:// that this particular Twitter streaming API requires doesn’t appear to work with urllib2.

I have tried to fetch the JSON from the following link: https://stream.twitter.com/1.1/statuses/filter.json?locations=-122.75,36.8,-121.75,37.8

By writing the following:

import urllib2
url = "https://stream.twitter.com/1.1/statuses/filter.json?locations=-122.75,36.8,-121.75,37.8"
print urllib2.urlopen(url).read()

The traceback, though, says I am not authorized; I presumed this is because of the HTTPS SSL connection, and research supported this. So I guess my question is what is the best way to go about getting the JSON from a link requiring such SSL connection? I also tried curl, but only get error 401 again.

Below is the traceback in its entirety.

Traceback (most recent call last): File “”, line 1, in
File
“/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py”,
line 126, in urlopen File
“/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py”,
line 400, in open File
“/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py”,
line 513, in http_response File
“/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py”,
line 438, in error File
“/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py”,
line 372, in _call_chain File
“/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py”,
line 521, in http_error_default urllib2.HTTPError: HTTP Error 401:
Unauthorized

Many thanks in advance.

Answer 1

As others mention, you need to authorize. Here’s how to do it:

See Twitter API with urllib2 in python

import urllib2
import base64

request = urllib2.Request( 'https://stream.twitter.com...' )
request.add_header( 'Authorization', 'Basic ' + base64.b64encode( username + ':' + password ) )
response = urllib2.urlopen( request )