Question

The following code works fine on my machine, but it is throwing an error at the line

soup = BeautifulSoup(html)

When it’s run on another machine. It’s parsing a list of active NBA players off of yahoo sports and storing their names and positions to a text file.

from bs4 import BeautifulSoup
import urllib2

'''
scraping the labeled data from yahoo sports
'''
def scrape(filename):
    base_url = "http://sports.yahoo.com/nba/players?type=position&c=NBA&pos="
    positions = ['G', 'F', 'C']
    players = 0

    with open(filename, 'w') as names:
        for p in positions:
            html = urllib2.urlopen(base_url + p).read()
            soup = BeautifulSoup(html) #throws the error!
            table = soup.find_all('table')[9]
            cells = table.find_all('td')

            for i in xrange(4, len(cells) - 1, 3):
                names.write(cells[i].find('a').string + '\t' + p + '\n')
                players += 1

    print "...success! %r players downloaded." % players

The error it throws is:

Traceback (most recent call last):
  File "run_me.py", line 9, in <module>
    scrapenames.scrape('namelist.txt')
  File "/Users/brapse/Downloads/bball/scrapenames.py", line 15, in scrape
    soup = BeautifulSoup(html)
  File "/usr/local/Cellar/python/2.6.5/lib/python2.6/site-packages/bs4/__init__.py", line 100, in __init__
    self._feed()
  File "/usr/local/Cellar/python/2.6.5/lib/python2.6/site-packages/bs4/__init__.py", line 113, in _feed
    self.builder.feed(self.markup)
  File "/usr/local/Cellar/python/2.6.5/lib/python2.6/site-packages/bs4/builder/_htmlparser.py", line 46, in feed
    super(HTMLParserTreeBuilder, self).feed(markup)
  File "/usr/local/Cellar/python/2.6.5/lib/python2.6/HTMLParser.py", line 108, in feed
    self.goahead(0)
  File "/usr/local/Cellar/python/2.6.5/lib/python2.6/HTMLParser.py", line 171, in goahead
    self.handle_charref(name)
  File "/usr/local/Cellar/python/2.6.5/lib/python2.6/site-packages/bs4/builder/_htmlparser.py", line 58, in handle_charref
    self.handle_data(unichr(int(name)))
ValueError: invalid literal for int() with base 10: 'xBB'

Answer 1

I believe it is a bug in the BS4 htmlparser code, it would crash on the » entity (stands for »), thinking that it should be in decimal. I suggest you update BeautifulSoup on that machine.