Question

Im new to python and in my code below: I have a crawler that recurses on new discovered links. After recursing on the root link, it seems like the program is stop after printing a couple of links, this should go on for a while, but its not. I am catching and printing exceptions but the program terminates successful, so im not really sure why its stopping.

from urllib import urlopen
from bs4 import BeautifulSoup

def crawl(url, seen):
    try:
    if any(url in s for s in seen):
       return 0
    html = urlopen(url).read()

    soup = BeautifulSoup(html)
    for tag in soup.findAll('a', href=True):
        str = tag['href']
        if 'http' in str:
        print tag['href']
        seen.append(str)
        print "--------------"
        crawl(str, seen)
    except Exception, e:
      print e
      return 0

def main ():
    print "$ = " , crawl("http://news.google.ca", [])


if __name__ == "__main__":
    main()

Answer 1

try:
    if any(url in s for s in seen):
       return 0

and then

seen.append(str)
print "--------------"
crawl(str, seen)

You append str to seen, and then call crawl with str and seen as arguments. Obviously your code exits. You have designed it in such a way.

A better way would be to crawl one page, add all the links found to a list that is to be crawled, and then keep crawling all the links in that list.

In simpler terms, instead of going for depth first crawling, you should do breadth first crawling.

Something like this should work.

from urllib import urlopen
from bs4 import BeautifulSoup

def crawl(url, seen, to_crawl):
    html = urlopen(url).read()
    soup = BeautifulSoup(html)
    seen.append(url)
    for tag in soup.findAll('a', href=True):
        str = tag['href']
        if 'http' in str:
            if url not in seen and url not in to_crawl:
                to_crawl.append(str)
                print tag['href']
                print "--------------"
    crawl(to_crawl.pop(), seen, to_crawl)

def main ():
    print "$ = " , crawl("http://news.google.ca", [], [])


if __name__ == "__main__":
    main()

Though you might wanna put a limit on the maximum depth or max number of urls it will crawl.