Question

I am following this video to get myself familiar with selenium. My code is

from selenium import webdriver
from selenium.webdriver.common.keys import Keys

from pyvirtualdisplay import Display
import os

chromedriver = "/usr/bin/chromedriver"
os.environ['webdriver.chrome.driver'] = chromedriver

display = Display(visible=0, size=(800,600))
display.start()

br = webdriver.Chrome(chromedriver)
br.get("http://www.google.com")

Now to print the results

q = br.find_element_by_name('q')
q.send_keys('python')
q.send_keys(Keys.RETURN)
print br.title

results = br.find_elements_by_class_name('g')
print results

for result in results:
    print result.text
    print "-"*140

The output I am getting is just python and when I try to print results it is [].

When I try the below code in chrome’s javascript console it works fine.

res = document.getElementsByClassName('g')[0]
<li class=​"g">​…​</li>​
res.textContent
"           Python Programming Language – Official Websitewww.python.org/Cached - SimilarShareShared on Google+. View the post.You +1'd this publicly. UndoHome page for Python, an interpreted, interactive, object-oriented, extensible programming language. It provides an extraordinary combination of clarity and ...CPython - Documentation - IDEs - GuiProgramming"

So, any idea why am I not getting any results with selenium+python.

Answer 1

Adding time.sleep(3) after q.send_keys(Keys.RETURN) seems to solve the problem. That’s because when you press Keys.RETURN, ajax starts working and when you try to collect the result, they aren’t on page yet. Selenium, AFAI has no stright way to determine whether the scripts like this have finished execution.

As I think, it would be more reliable to do

br.get("http://www.google.com/search?q=python")
results = br.find_elements_by_class_name('g')