Question

I tried all I can do to encode the page then using BeautifulSoup. However, when I run, it shows the unicode results. Can anyone help me how to encode under BeautifulSoup

my code:

import httplib
import urllib
import urllib2
from BeautifulSoup import BeautifulSoup
import HTMLParser


headers={
'Host': 'digitalvita.pitt.edu',
'Connection': 'keep-alive',
'Origin': 'https://digitalvita.pitt.edu',
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1',
'Content-type': 'application/x-www-form-urlencoded; charset=UTF-8',
'Accept': 'text/javascript, text/html, application/xml, text/xml, */*',
'Referer': 'https://digitalvita.pitt.edu/index.php',
'Accept-Encoding': 'gzip,deflate,sdch',
'Accept-Language': 'en-US,en;q=0.8',
'Accept-Charset': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3',
'Cookie': 'PHPSESSID=lvetilatpgs9okgrntk1nvn595'
}

data={
'action':'search',
'xdata':'<search id="1"><context type="all" /><results><ordering>familyName</ordering><pagesize>100000</pagesize><page>1</page></results><terms><name>d</name><school>All</school></terms></search>',
'request':'search'
}

data = urllib.urlencode(data)
print data
req = urllib2.Request('https://digitalvita.pitt.edu/dispatcher.php', data, headers)
response = urllib2.urlopen(req)
the_page = response.read()

htmlCodes = [
    ['&', '&amp;'],
    ['<', '&lt;'],
    ['>', '&gt;'],
    ['"', '&quot;'],
]
htmlCodesReversed = htmlCodes[:]
htmlCodesReversed.reverse()

def htmlEncode(s, codes=htmlCodes):
    """ Returns the HTML encoded version of the given string. This is useful to
        display a plain ASCII text string on a web page."""
    for code in codes:
        s = s.replace(code[1], code[0])
    return s
s=htmlEncode(the_page,codes=htmlCodes)

h = HTMLParser.HTMLParser()
s=h.unescape(s)

s.encode("utf-8")

soup=BeautifulSoup(s,convertEntities=BeautifulSoup.HTML_ENTITIES)
print soup

The simple results is like:

 <a href="#local" onclick="dvSearch.ToggleInterests(141432);"><span class="iToggle" id="toggle_141432">more...</span></a></span></div></td></tr></table></div><div><table width="100%" cellspacing="5" cellpadding="0"><tr valign="top"><td><img width="70" height="70" src="http://digitalvita.pitt.edu/digital-vitaUI.profileimages/na.jpg" /></td><td width="99%"><div><span class="name"> Znati, Taieb</span><span class="email"> (<a href="mailto:[e-mail]">[e-mail]</a>) </span></div><div class="professionalPosition">Computer Science, University of Pittsburgh</div></td></tr></table></div><div><table width="100%" cellspacing="5" cellpadding="0"><tr valign="top"><td><img width="70" height="70" src="http://digitalvita.pitt.edu/digital-vitaUI.profileimages/na.jpg" /></td><td width="99%"><div><span class="name"> Zoffer, H</span><span class="email"> (<a href="mailto:[e-mail]">[e-mail]</a>) </span></div><div class="professionalPosition">"KGSB-Dean, Office of", University of Pittsburgh</div></td></tr></table></div><div><table width="100%" cellspacing="5" cellpadding="0"><tr valign="top"><td><img width="70" height="70" src="http://digitalvita.pitt.edu/digital-vitaUI.profileimages/na.jpg" /></td><td width="99%"><div><span class="name"> Zorn, Kristin</span><span class="email"> (<a href="mailto:[e-mail]">[e-mail]</a>) </span></div></td></tr></table></div><div><table width="100%" cellspacing="5" cellpadding="0"><tr valign="top"><td><img width="70" height="70" src="http://digitalvita.pitt.edu/digital-vitaUI.profileimages/na.jpg" /></td><td width="99%"><div><span class="name"> Zou, Chunbin</span><span class="email"> (<a href="mailto:[e-mail]">[e-mail]</a>) </span></div><div class="researchInterest"><b>Research Interests: </b>fatty liver disease; tyrosine kinase receptor; proteasome endopeptidase complex; phosphatidylcholines; trypanosome; Fas; ubiquitin; pulmonary surfactants; HGF/Met</div></td></tr></table></div><div><table width="100%" cellspacing="5" cellpadding="0"><tr valign="top"><td><img width="70" height="70" src="http://digitalvita.pitt.edu/digital-vitaUI.profileimages/na.jpg" /></td><td width="99%"><div><span class="name"> Zou, Xiuying</span><span class="email"> (<a href="mailto:[e-mail]">[e-mail]</a>) </span></div></td></tr></table></div><div><table width="100%" cellspacing="5" cellpadding="0"><tr valign="top"><td><img width="70" height="70" src="http://digitalvita.pitt.edu/digital-vitaUI.profileimages/na.jpg" /></td><td width="99%"><div><span class="name"> Zrust, Marilyn</span><span class="email"> (<a href="mailto:[e-mail]">[e-mail]</a>) </span></div><div class="professionalPosition">Clinical Instructor, Acute/Tertiary Care, University of Pittsburgh School of Nursing</div></td></tr></table></div><div><table width="100%" cellspacing="5" cellpadding="0"><tr valign="top"><td><img width="70" height="70" src="http://digitalvita.pitt.edu/digital-vitaUI.profileimages/na.jpg" /></td><td width="99%"><div><span class="name"> Zubieta, Juan</span><span class="email"> (<a href="mailto:[e-mail]">[e-mail]</a>) </span></div></td></tr></table></div><div><table width="100%" cellspacing="5" cellpadding="0"><tr valign="top"><td><img width="70" height="70" src="http://digitalvita.pitt.edu/digital-vitaUI.profileimages/na.jpg" /></td><td width="99%"><div><span class="name"> Zuccoli, Giulio</span><span class="email"> (<a href="mailto:[e-mail]">[e-mail]</a>) </span></div></td></tr></table></div><div><table width="100%" cellspacing="5" cellpadding="0"><tr valign="top"><td><img width="70" height="70" src="http://digitalvita.pitt.edu/digital-vitaUI.profileimages/na.jpg" /></td><td width="99%"><div><span class="name"> Zuckerman, Daniel</span><span class="email"> (<a href="mailto:[e-mail]">[e-mail]</a>) </span></div><div class="professionalPosition">Computational Biology, University of Pittsburgh</div><div class="researchInterest"><b>Research Interests: </b>structural biology; stochastic processes; computer simulation; coarse-grained models; protein dynamics and fluctuations; models, theoretical</div></td></tr></table></div><div><table width="100%" cellspacing="5" cellpadding="0"><tr valign="top"><td><img width="70" height="70" src="http://digitalvita.pitt.edu/digital-vitaUI.profileimages/na.jpg" /></td><td width="99%"><div><span class="name"> Zuckoff, Allan</span><span class="email"> (<a href="mailto:[e-mail]">[e-mail]</a>) </span></div><div class="professionalPosition">Psychology, University of Pittsburgh</div></td></tr></table></div><div><table width="100%" cellspacing="5" cellpadding="0"><tr valign="top"><td><img width="70" height="70" src="http://digitalvita.pitt.edu/digital-vitaUI.profileimages/na.jpg" /></td><td width="99%"><div><span class="name"> Zuckoff, Allan</span><span class="email"> (<a href="mailto:[e-mail]">[e-mail]</a>) </span></div><div class="professionalPosition">Psychiatry, University of Pittsburgh</div><div class="researchInterest"><b>Research Interests: </b>psychotherapy; substance-related disorders; motivational interviewing; grief treatment ; diagnosis, dual (psychiatry); treatment adherence; patient compliance; traumatic grief and substance abuse</div></td></tr></table></div><div><table width="100%" cellspacing="5" cellpadding="0"><tr valign="top"><td><img width="70" height="70" src="http://digitalvita.pitt.edu/digital-vitaUI.profileimages/na.jpg" /></td><td width="99%"><div><span class="name"> Zukor, Tevya</span><span class="email"> (<a href="mailto:[e-mail]">[e-mail]</a>) </span></div></td></tr></table></div><div><table width="100%" cellspacing="5" cellpadding="0"><tr valign="top"><td><img width="70" height="70" src="http://digitalvita.pitt.edu/digital-vitaUI.profileimages/na.jpg" /></td><td width="99%"><div><span class="name"> Zuley, Margarita</span><span class="email"> (<a href="mailto:[e-mail]">[e-mail]</a>) </span></div></td></tr></table></div><div><table width="100%" cellspacing="5" cellpadding="0"><tr valign="top"><td><img width="70" height="70" src="http://digitalvita.pitt.edu/digital-vitaUI.profileimages/na.jpg" /></td><td width="99%"><div><span class="name"> Zunino, Paolo</span><span class="email"> (<a href="mailto:[e-mail]">[e-mail]</a>) </span></div><div class="professionalPosition">Mech Eng and Materials Sci, University of Pittsburgh</div></td></tr></table></div><div><table width="100%" cellspacing="5" cellpadding="0"><tr valign="top"><td><img width="70" height="70" src="http://digitalvita.pitt.edu/digital-vitaUI.profileimages/na.jpg" /></td><td width="99%"><div><span class="name"> Zureikat, Amer</span><span class="email"> (<a href="mailto:[e-mail]">[e-mail]</a>) </span></div></td></tr></table></div><div><table width="100%" cellspacing="5" cellpadding="0"><tr valign="top"><td><img width="70" height="70" src="http://digitalvita.pitt.edu/digital-vitaUI.profileimages/na.jpg" /></td><td width="99%"><div><span class="name"> Zutter, Chad</span><span class="email"> (<a href="mailto:[e-mail]">[e-mail]</a>) </span></div><div class="professionalPosition">KGSB-Business Admin, University of Pittsburgh</div></td></tr></table></div><div><table width="100%" cellspacing="5" cellpadding="0"><tr valign="top"><td><img width="70" height="70" src="http://digitalvita.pitt.edu/digital-vitaUI.profileimages/na.jpg" /></td><td width="99%"><div><span class="name"> Zyczynski, Halina</span><span class="email"> (<a href="mailto:[e-mail]">[e-mail]</a>) </span></div><div class="professionalPosition">Obstetrics, Gynecology and Reproductive Sciences, University of Pittsburgh</div><div class="researchInterest"><b>Research Interests: </b>pelvic floor reconstruction; rectocele; uterine prolapse; sacralcolpopexy; bladder diseases; colpocleisis; pelic organ prolapse; urinary incontinence</div></td></tr></table></div>

]]>

Answer 1

It looks like the problem is that you’re mixing up character sets.

The first thing I’d do is change your Accept-Charset so you only accept utf-8.

'Accept-Charset': 'utf-8;q=0.7,*;q=0.3',

Next, the result of response.read() is an 8-bit string, which you have to decode. Since we now know that it’s utf-8, you can do this:

the_page = response.read().decode('utf-8')

With those two changes, when I run your script, the same fragment comes back as:

 … Self Care</span>
                                            <a href="#local" onclick="dvSearch.ToggleInterests(…

No more garbage Unicode characters.

Of course this only works because the server is willing to return utf-8. For a more general case, where you have some servers that can only do utf-8 and others that can only do Latin-1, you need to do something a bit more complicated. Leave the Accept-Charset header alone, and then change the read to look at the response headers. Something like this:

response = urllib2.urlopen(req)
charset = response.info().getencoding()
the_page = response.read().decode(charset)

There are many badly-configured servers that won’t actually return a charset, even when they aren’t returning pure 7-bit ASCII. In that case, you need to either examine what the server returns and hardcode the right answer, or write code to try to detect the proper charset on the fly. Hopefully you’ll never run into this situation…