Question

I am working on a website scraping project using Selenium in Python. When I open the homepage through a browser, it opens properly.

But, when I try to open the webpage through webdriver() in Selenium, it opens a completely different page.

I think, it is able to detect the user-agent( not sure what it is called) and is able to check the properties of the browser or something.

Is it possible to pass the properties though the webdriver() so that the right homepage is loaded.

Thanks

Answer 1

Changing the user agent in the python version of webdriver is done by altering your browser’s profile. I have only done this for webdriver.Firefox() by passing a profile parameter. You need to do the following:

from selenium import webdriver
profile = webdriver.FirefoxProfile()
profile.set_preference("general.useragent.override","your_user_agent_string")
driver=webdriver.Firefox(profile)

Every time you wish to change the user agent you will need to restart your web browser (i.e. call driver=webdriver.Firefox(profile) again)

If you are unsure to what your user agent string is do a search for “what is my user agent” on a browser that displays the page properly and just copy and paste that one.

Hope that sorts it.