Question

I am trying to check if a certain word is on a page for many sites. The script runs fine for say 15 sites and then it stops.

UnicodeDecodeError: ‘utf8’ codec can’t decode byte 0x96 in position 15344: invalid start byte

I did a search on stackoverflow and found many issues on it but I can’t seem to understand what went wrong in my case.

I would like to either solve it or if there is an error skip that site. Pls advice how I can do this as I am new and the below code itself has taken me a day to write. By the way the site which the script halted on was http://www.homestead.com

filetocheck = open("bloglistforcommenting","r")
resultfile = open("finalfile","w")

for countofsites in filetocheck.readlines():
        sitename = countofsites.strip()
        htmlfile = urllib.urlopen(sitename)
        page = htmlfile.read().decode('utf8')
        match = re.search("Enter your name", page)
        if match:
            print "match found  : " + sitename
            resultfile.write(sitename+"\n")

        else:
            print "sorry did not find the pattern " +sitename

print "Finished Operations"

As per Mark’s comments I changed the code to implement beautifulsoup

htmlfile = urllib.urlopen("http://www.homestead.com")
page = BeautifulSoup((''.join(htmlfile)))
print page.prettify() 

now I am getting this error

page = BeautifulSoup((''.join(htmlfile)))
TypeError: 'module' object is not callable

I am trying their quick start example from http://www.crummy.com/software/BeautifulSoup/documentation.html#Quick%20Start. If I copy paste it then the code works fine.

I FINALLY got it to work. Thank you all for your help. Here is the final code.

import urllib
import re
from BeautifulSoup import BeautifulSoup

filetocheck = open("listfile","r")

resultfile = open("finalfile","w")
error ="for errors"

for countofsites in filetocheck.readlines():
        sitename = countofsites.strip()
        htmlfile = urllib.urlopen(sitename)
        page = BeautifulSoup((''.join(htmlfile)))  
        pagetwo =str(page) 
        match = re.search("Enter YourName", pagetwo)
        if match:
            print "match found  : " + sitename
            resultfile.write(sitename+"\n")

        else:
            print "sorry did not find the pattern " +sitename

print "Finished Operations"

Answer 1

Many web pages are encoded incorrectly. For parsing HTML try BeautifulSoup as it can handle many types of incorrect HTML that are found in the wild.

Beautiful Soup is a Python HTML/XML parser designed for quick
turnaround projects like screen-scraping. Three features make it
powerful:

1. Beautiful Soup won’t choke if you give it bad markup. It yields a
   parse tree that makes approximately as much sense as your original
   document. This is usually good enough to collect the data you need and
   run away.

2. Beautiful Soup provides a few simple methods and Pythonic
   idioms for navigating, searching, and modifying a parse tree: a
   toolkit for dissecting a document and extracting what you need. You
   don’t have to create a custom parser for each application.

3. Beautiful
   Soup automatically converts incoming documents to Unicode and outgoing
   documents to UTF-8. You don’t have to think about encodings, unless
   the document doesn’t specify an encoding and Beautiful Soup can’t
   autodetect one. Then you just have to specify the original encoding.

Emphasis mine.