Question

In python, I want to send a request to a url which will return some information to me. The problem is if I try to access the url from the browser, a popup box appears and asks for a username and password. I have a username and password for this url however, I don’t know how to make python automatically complete these fields to access the URL.

This is my code that returns a 401 error:

bing = "https://api.datamarket.azure.com/Data.ashx/Bing/SearchWeb/Web?Query=%
(term)s&$top=50&$format=json" %__function()
results = urllib2.urlopen(bing).read()

EDIT: Just for clarification, the function function returns a variable called term which is the term being queried in the URL

Answer 1

You’re most likely facing HTTP basic authentication. This can be done in urllib2 but it’s a bit tiresome. The instructions on how to do this are here:

http://www.voidspace.org.uk/python/articles/urllib2.shtml#id5

But if you haven’t already done much work with urllib2, now is a good time to switch to the requests library. It is much more user friendly – and to do what you ask is one line call – unlike the messy way in urllib2.

requests.get('https://api.github.com/user', auth=('user', 'pass'))

More details on how requests library can be found at: http://docs.python-requests.org/en/latest/user/quickstart/#basic-authentication

To install requests library you only need to do:

sudo pip install requests

If you’re on windows you should drop the sudo part.