Question

I have to load some url with cyrillic symbols. My script should work with this:

http://wincode.org/%D0%BF%D1%80%D0%BE%D0%B3%D1%80%D0%B0%D0%BC%D0%BC%D0%B8%D1%80%D0%BE%D0%B2%D0%B0%D0%BD%D0%B8%D0%B5/

If I’ll use this in browser it would replaced into normal symbols, but urllib code fails with 404 error. How to decode correctly this url?

---

When I’m using that url directly in code, like address = ‘that address’, it works perfect. But I used parsing page for getting this url. I have a list of urls which contents cyrillic. Maybe they have uncorrect encoding? Here is more code:

requestData = urllib2.Request( %SOME_ADDRESS%, None,  {"User-Agent": user_agent})
requestHandler = pageHandler.open(requestData)

pageData = requestHandler.read().decode('utf-8')
soupHandler = BeautifulSoup(pageData)

topicLinks = []
for postBlock in soupHandler.findAll('a', href=re.compile('%SOME_REGEXP%')):
    topicLinks.append(postBlock['href'])

postAddress = choice(topicLinks)

postRequestData = urllib2.Request(postAddress, None,  {"User-Agent": user_agent})
postHandler = pageHandler.open(postRequestData)
postData = postHandler.read()

  File "/usr/lib/python2.6/urllib2.py", line 518, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 404: Not Found

Answer 1

I have a list of urls which contents cyrillic.

OK, if it contains raw (not %-encoded) Cyrillic characters that’s not like the example, and in fact it isn’t a URL at all.

An address with non-ASCII characters in it is known as an IRI. IRIs shouldn’t be used in an HTML link, but browsers tend to fix up these mistakes.

To convert an IRI to a URI which you can then open with urllib, you have to:

1. encode non-ASCII characters in the hostname part using Punycode (IDNA).

2. encode non-ASCII characters in rest of the IRI to UTF-8 bytes and URL-encode them (resulting in %D0%BF... like in the example URL).

an example implementation.