Question

I’m having an issue with testing this BeautifulSoup crawler. Forgive me if there is some obvious mistake, since this is my third hour into Python. I have this code below…

def huffpost_crawl():
 article_list = []
 DOMAIN = 'huffingtonpost.com'
 huff_soup = BeautifulSoup(urllib2.urlopen("http://www.huffingtonpost.com").read())
 news_list = huff_soup.find_all("div", {"class", "snp_most_popular_entry"})[0]
 for news in news_list[0]:
    title = news('div', {'class', 'snp_most_popular_entry_desc'})[0].a.get_text()
    full_url = news('div', {'class', 'snp_most_popular_entry_image'}).a["href"]
    blurb = ""
    thumb_url = news('div', {'class', 
   'snp_most_popular_entry_image'}).a.img["longdesc"]


 print title

huffpost_crawl()

When I run pythong test.py in my terminal, I’m returned with…

Traceback (most recent call last):
  File "test.py", line 21, in <module>
  huffpost_crawl()
File "test.py", line 11, in huffpost_crawl
  for news in news_list[0]:
File "/usr/local/lib/python2.7/site-packages/bs4/element.py", line 879, in __getitem__
  return self.attrs[key]
KeyError: 0

Answer 1

Here’s the problem:

news_list = huff_soup.find_all("div", {"class", "snp_most_popular_entry"})[0]
for news in news_list[0]:

Just remove one of those two [0] bits and the problem (or at least this problem—I can’t guarantee that the rest of your code does what you want) will go away.

I won’t explain why the code was wrong, because you really need to learn to debug your code and figure this stuff out yourself.

Start by doing this in an interactive interpreter:

>>> huff_soup = BeautifulSoup(urllib2.urlopen("http://www.huffingtonpost.com").read())
>>> news_list = huff_soup.find_all("div", {"class", "snp_most_popular_entry"})

See what that returns—what shape is it, and how do you get to the part you want interactively? Once you know that, it should be obvious how to do it in your script.

Even in cases where things are too complex to play with interactively, you can log stuff with print statements, run in the debugger, etc. Don’t just blindly stare at code that doesn’t work and say “Why doesn’t it work?”, or post the code somewhere and ask someone else why it doesn’t work, or you’ll never learn anything.