Question

I want to screen scrape a web site having multiple pages. These pages are loaded dynamically without changing the URL. Hence I’m using selenium to screen scrape it. But I’m getting an exception for this simple program.

import re
from contextlib import closing
from selenium.webdriver import Firefox 

url="http://www.samsung.com/in/consumer/mobile-phone/mobile-phone/smartphone/"

with closing(Firefox()) as browser:
    n = 2
    link = browser.find_element_by_link_text(str(n))
    link.click()
    #web_page=browser.page_source
    #print type(web_page)

Error is as follows

raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: u'Unable to locate element: {"method":"link text","selector":"2"}' ; Stacktrace: Method FirefoxDriver.prototype.findElementInternal_ threw an error in file:///tmp/tmpMJeeTr/extensions/[e-mail]/components/driver_component.js 

Is it the problem with the url given or with the firefox browser.
Would be great help if someone helped me.

Answer 1

I think your main issue is that the page itself takes a while to load, and you are immediately trying to access that link (which likely hasn’t yet rendered, hence the stack trace). One thing you can try is using an Implicit Wait 1 with your browser, which will tell the browser to wait for a certain period of time for elements to appear before timing out. In your case, you could try the following, which would wait for up to 10 seconds while polling the DOM for a particular item (in this case, the link text 2):

browser.implicitly_wait(10)
n = 2
link = browser.find_element_by_link_text(str(n))
link.click()
#web_page=browser.page_source
#print type(web_page)