Question

I recently asked this question about encoding Hindi characters in BeautifulSoup.
The answers to that question did solve that issue, however I have another problem.

My code is :

import urllib2
from bs4 import BeautifulSoup

htmlUrl = "http://archives.ndtv.com/articles/2012-01.html"
FileName = "NDTV_2012_01.txt"

fptr = open(FileName, "w")
fptr.seek(0)

page = urllib2.urlopen(htmlUrl)
soup = BeautifulSoup(page, from_encoding="UTF-8")

li = soup.findAll( 'li')
for link_tag in li:
  hypref = link_tag.find('a').contents[0]
  strhyp = hypref.encode('utf-8')
  fptr.write(strhyp)
  fptr.write("\n")

And I get an error

Traceback (most recent call last):
File "./ScrapeTemplate.py", line 29, in <module>
hypref = link_tag.find('a').contents[0]
IndexError: list index out of range

It seems to work when I substitute print strhyp instead of fptr.write(). How do I fix this?

EDIT: There was a mistake in the code which I didn’t spot. Fixed it, but I’m still getting the same error.

Answer 1

Your code is tripping over the links at the bottom of the page. Skip over these:

for link_tag in li:
  contents = link_tag.find('a').contents
  if len(contents) > 0:
    hypref = contents[0]
    strhyp = hypref.encode('utf-8')
    fptr.write(strhyp)
    fptr.write("\n")