Question

I am using the following code to open a url and retrieve it’s response :

def get_issue_report(query):
    request = urllib2.Request(query)
    response = urllib2.urlopen(request)
    response_headers = response.info()
    print response.read()

The response I get is as follows :

<?xml version='1.0' encoding='UTF-8'?><entry xmlns='http://www.w3.org/2005/Atom' xmlns:gd='http://schemas.google.com/g/2005' xmlns:issues='http://schemas.google.com/projecthosting/issues/2009' gd:etag='W/&quot;DUUFQH47eCl7ImA9WxBbFEg.&quot;'><id>http://code.google.com/feeds/issues/p/chromium/issues/full/2</id><published>2008-08-30T16:00:21.000Z</published><updated>2010-03-13T05:13:31.000Z</updated><title>Testing if chromium id works</title><content type='html'>&lt;b&gt;What steps will reproduce the problem?&lt;/b&gt;
&lt;b&gt;1.&lt;/b&gt;
&lt;b&gt;2.&lt;/b&gt;
&lt;b&gt;3.&lt;/b&gt;

&lt;b&gt;What is the expected output? What do you see instead?&lt;/b&gt;


&lt;b&gt;Please use labels and text to provide additional information.&lt;/b&gt;
 </content><link rel='replies' type='application/atom+xml' href='http://code.google.com/feeds/issues/p/chromium/issues/2/comments/full'/><link rel='alternate' type='text/html' href='http://code.google.com/p/chromium/issues/detail?id=2'/><link rel='self' type='application/atom+xml' href='https://code.google.com/feeds/issues/p/chromium/issues/full/2'/><author><name>[e-mail]</name><uri>/u/@VBJVRVdXDhZCVgJ%2FF3tbUV5SAw%3D%3D/</uri></author><issues:closedDate>2008-08-30T20:48:43.000Z</issues:closedDate><issues:id>2</issues:id><issues:label>Type-Bug</issues:label><issues:label>Priority-Medium</issues:label><issues:owner><issues:uri>/u/[e-mail]/</issues:uri><issues:username>[e-mail]</issues:username></issues:owner><issues:stars>4</issues:stars><issues:state>closed</issues:state><issues:status>Invalid</issues:status></entry>

I would like to get rid of the characters like &lt, &gt etc. I tried using

response.read().decode(‘utf-8’)

but this doesn’t help much.

Just in case, the response.info() prints the following :

Content-Type: application/atom+xml; charset=UTF-8; type=entry
Expires: Fri, 01 Jul 2011 11:15:17 GMT
Date: Fri, 01 Jul 2011 11:15:17 GMT
Cache-Control: private, max-age=0, must-revalidate, no-transform
Vary: Accept, X-GData-Authorization, GData-Version
GData-Version: 1.0
ETag: W/"DUUFQH47eCl7ImA9WxBbFEg."
Last-Modified: Sat, 13 Mar 2010 05:13:31 GMT
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Server: GSE
Connection: close

Here’s the URL : https://code.google.com/feeds/issues/p/chromium/issues/full/2

Answer 1

from HTMLParser import HTMLParser
import urllib2


query="http://code.google.com/feeds/issues/p/chromium/issues/full/2"

def get_issue_report(query):
    request = urllib2.Request(query)
    response = urllib2.urlopen(request)
    response_headers = response.info()
    return response.read()

s = get_issue_report(query)

p = HTMLParser()

print p.unescape(s)

p.close()