Question

i am using BeautifulSoup module to select all href from html by this way:

def extract_links(html):
  soup = BeautifulSoup(html)
  anchors = soup.findAll('a')
  print anchors
  links = []
  for a in anchors:
    links.append(a['href'])
  return links

but sometime it failed by this error message:

Traceback (most recent call last):
File "C:\py\main.py", line 33, in <module>
urls = extract_links(page)
File "C:\py\main.py", line 11, in extract_links
links.append(a['href'])
File "C:\py\BeautifulSoup.py", line 601, in __getitem__
return self._getAttrMap()[key]
KeyError: 'href'

Answer 1

Not all anchor tags will have an href attribute. You should check that the anchor has an href before you try to access that attribute.

if a.has_key('href')
  links.append(a['href'])

After checking some comments here, I think this is the most pythonic way of handling this case.