Question

when executing a python script which scrapes historical tweets
I after about 2 to 3 min. get following error!!!

I really dont know what this means

and how I could prevent this from happening

I am defiantly in my rate limit because before executing each scraping loop I check and it returns as true!!!

thanks for your help

Traceback (most recent call last):
  File "twitter.py", line 13, in <module>
openurl = urllib.urlopen("https://api.twitter.com/1/statuses/user_timeline.json?include_entities=true&contributor_details&include_rts=true&screen_name="+user+"&count=3600")
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 84, in urlopen
return opener.open(url)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 205, in open
return getattr(self, name)(url)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 449, in open_https
return self.http_error(url, fp, errcode, errmsg, headers)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 369, in http_error
result = method(url, fp, errcode, errmsg, headers)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 691, in http_error_401
errcode, errmsg, headers)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 379, in http_error_default
raise IOError, ('http error', errcode, errmsg, headers)
IOError: ('http error', 401, 'Unauthorized', <httplib.HTTPMessage instance at 0x1005ca9e0>)

the script works perfectly but then after 2-3 min it always breaks….

Answer 1

Twitter api has a 150 query per hour limit
https://dev.twitter.com/docs/rate-limiting
Try to query more than one tweet in one query.
I think an explanation comes with your 401 response
try this

try:
    response = urllib2.urlopen.....
except urllib2.HTTPError as e:
    error = e.read() # this will be your error message
    print error