Question

I am using beautiful soup and I am writing a crawler and have the following code in it:

  print soup.originalEncoding
                #self.addtoindex(page, soup)


                links=soup('a')
            for link in links:

                if('href' in dict(link.attrs)):                   
                    link['href'].replace('..', '')
                    url=urljoin(page, link['href'])
                    if url.find("'") != -1:
                        continue
                    url = url.split('?')[0]
                    url = url.split('#')[0]
                    if url[0:4] == 'http':
                        newpages.add(url)
        pages = newpages

The link['href'].replace('..', '') is supposed to fix links that come out as ../contact/orderform.aspx, ../contact/requestconsult.aspx, etc. However, it is not working. Links still have the leading “..” Is there something I am missing?

Answer 1

string.replace() returns the string with the replaced values. It doesn’t modify the original so do something like this:

link['href'] = link['href'].replace("..", "")