Question

I have a website which generates more products when I scroll down. Unlike other websites, there is nothing found in the firebug console. So, I am using selenium to simulate a browser. I have made it working but with firefox driver. But, since I am hosting my web server which runs on command line, I am using HTMLUNIT. Can someone tell me how to scroll pages using HTMLUNIT? Here is the code till now:

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time

browser = webdriver.Remote("http://127.0.0.1:4444/wd/hub",desired_capabilities=webdriver.DesiredCapabilities.HTMLUNITWITHJS)
browser.get("http://www.somewebsite.com/")
x = browser.find_elements_by_xpath("//div[@id='containeriso3']/div/a[1]")
hrefs = [i.get_attribute('href') for i in x]
print len(hrefs)
time.sleep(2)
browser.execute_script("scroll(0, 2500);")
time.sleep(2)
x = browser.find_elements_by_xpath("//div[@id='containeriso3']/div/a[1]")
hrefs = [i.get_attribute('href') for i in x]
print len(hrefs)

Thank you.

Answer 1

You can use JavaScript to scroll. From the docs:

You can use the execute_script method to execute javascript on the loaded page.
So, you can call the JavaScript API to scroll to the bottom or any other position of a page.

Here is an example to scroll to the bottom of a page:

driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

The window object in DOM has a scrollTo method to scroll to any position of an opened window.
The scrollHeight is a common property for all elements. The document.body.scrollHeight will give the height
of the entire body of the page.