Question

I’m trying to parse the HTML of a webpage that requires being logged in. I can get the HTML of a webpage using this script:

from urllib2 import urlopen
from BeautifulSoup import BeautifulSoup
import re

webpage = urlopen ('https://www.example.com')
soup = BeautifulSoup (webpage)
print soup
#This would print the source of example.com

But trying to get the source of a webpage that I’m logged into proves to be more difficult.
I tried replacing the (‘https://www.example.com’) with (‘https://user:[e-mail]’) but I got an Invalid URL error.

Anyone know how I could do this?
Thanks in advance.

Answer 1

Selenium WebDriver ( http://seleniumhq.org/projects/webdriver/ ) might be good for your needs here. You can log in to the page and then print the contents of the HTML. Here’s an example:

from selenium import webdriver

# initiate
driver = webdriver.Firefox() # initiate a driver, in this case Firefox
driver.get("http://example.com") # go to the url

# locate the login form
username_field = driver.find_element_by_name(...) # get the username field
password_field = driver.find_element_by_name(...) # get the password field

# log in
username_field.send_keys("username") # enter in your username
password_field.send_keys("password") # enter in your password
password_field.submit() # submit it

# print HTML
html = driver.page_source
print html