Question

I am writing a basic screen scraping script using Mechanize and BeautifulSoup (BS) in Python. However, the problem I am running into is that for some reason the requested page does not download correctly every time. I am concluding this because when searching the downloaded pages using BS for present tags, I get an error. If I download the page again, it works.

Hence, I would like to write a small function that checks to see if the page has correctly downloaded and re-download if necessary (I could also solve it by figuring out what goes wrong, but that is probably too advanced for me). My question is how would I go about checking to see if the page has been downloaded correctly?

Answer 1

You can just check for a tag you expect to be there, and if it fails, repeat the download.

page = BeautifulSoup(page)

while page.body = None:
    #redownload the page
    page = BeautifulSoup(page)
#now you can use the data