Question

I have been running a cron job on Google App Engine for over a month now without any issues. The job does a variety of things, one being that it uses urllib2 to make a call to retrieve a json response from Reddit as well as a few other sites. About two weeks ago I started seeing errors when invoking Reddit, but no errors when invoking the other sites. The error I am receiving is HTTP error 429.

I have tried executing the same code outside of Google App Engine and do not have any issues. I tried using urlFetch, but receive the same error.

You can see the error when using the app engine’s interactive shell with the following code.

import urllib2
data = urllib2.urlopen('http://www.reddit.com/r/Music/.json', timeout=60)

Edit: Not sure why it always fails for me and not someone else. This is the error that I receive:

>>> import urllib2
>>> data = urllib2.urlopen('http://www.reddit.com/r/Music/.json', timeout=60)
Traceback (most recent call last):
  File "/base/data/home/apps/s~shell-27/1.356011914885973647/shell.py", line 267, in get
    exec compiled in statement_module.__dict__
  File "<string>", line 1, in <module>
  File "/base/python27_runtime/python27_dist/lib/python2.7/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/base/python27_runtime/python27_dist/lib/python2.7/urllib2.py", line 400, in open
    response = meth(req, response)
  File "/base/python27_runtime/python27_dist/lib/python2.7/urllib2.py", line 513, in http_response
    'http', request, response, code, msg, hdrs)
  File "/base/python27_runtime/python27_dist/lib/python2.7/urllib2.py", line 438, in error
    return self._call_chain(*args)
  File "/base/python27_runtime/python27_dist/lib/python2.7/urllib2.py", line 372, in _call_chain
    result = func(*args)
  File "/base/python27_runtime/python27_dist/lib/python2.7/urllib2.py", line 521, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 429: Unknown

similar code running outside of app engine with no problem:

print urllib2.urlopen('http://www.reddit.com/r/Music/.json').read()

At first I thought it had to do with a timeout problem since it was originally working, but since there is not a timeout error but a the strange HttpError code, I’m not sure.
Any ideas?

Answer 1

Reddit rate limits the api pretty severely for the default user agent for the python shell. You need to set a unique user agent with your reddit username in it, like this:

User-Agent: super happy flair bot by /u/spladug

More info about the reddit api here https://github.com/reddit/reddit/wiki/API.