Question

I have got URLs stored in a list which are supposed to be passed to urllib2. However, urllib doesn’t seem to like this very much and I just cannot see why!

here is what I have got:

url = list[1]
response = urllib2.urlopen(url)
html = response.read()

The URL is a google maps Directions Web API URL of the kind:

http://maps.googleapis.com/maps/api/directions/json?origin=[origin]&destination=[destination]&waypoints=optimize:true|[waypoint1]|[waypoint2]&sensor=false

Now, if I try to run this, the retrieved html always looks something like this:

{
"routes" : [],
"status" : "INVALID_REQUEST"
}

Indicating that something is wrong with the passed URL. If however, I take the URL and assign it directly, like so:

url = "http://maps.googleapis.com/maps/api/directions/json?origin=[origin]&destination=[destination]&waypoints=optimize:true|[waypoint1]|[waypoint2]&sensor=false"
response = urllib2.urlopen(url)
html = response.read()

The result will happily come through with the (for me) essential end part looking like this:

         "warnings" : [],
         "waypoint_order" : [ 2, 0, 7, 5, 6, 4, 3, 1 ]
      }
   ],
   "status" : "OK"
}

My (hopefully not too stupid) question is therefore: why does urllib do its job when the URL is assigned directly but not if it comes from a list??

Thank you very much for your help,
J

PS: Is there a reason why the forum-software always cuts off my ‘Hi all’ greeting?

Answer 1

I have finally resolved the issue!

Was pretty easy once I stepped back for a moment and thought about it again:
I did get an answer from google’s Web API which means that urllib2 did pass something. However, it must have been sending something that didn’t make any sense to the API backend. The URL containing German addresses I lucky-guessed the problem being caused by umlauts.

So I simply passed my URL through a function replacing each umlaut with its non-umlaut alternative and suddenly everything seemed to work fine.

If anybody comes accross a similar issue, here’s how I solved it:

# Function for replacing all umlauts
def replaceUmlauts(text):
    dic = {'Ä':'Ae', 'ä':'ae', 'Ö':'Oe', 'ö':'oe', 'Ü':'Ue', 'ü':'ue', 'ß':'ss'}
    for i, j in dic.iteritems():
        text = text.replace(i, j)
    return text

Then simply use

response = urllib2.urlopen(replaceUmlauts(url))