Question

New to programming and figured out how to navigate to where I need to go using Selenium. I’d like to parse the data now but not sure where to start. Can someone hold my hand a sec and point me in the right direction?

Any help appreciated –

Answer 1

Assuming you are on the page you want to parse, Selenium stores the source HTML in the driver’s page_source attribute. You would then load the page_source into BeautifulSoup as follows:

from bs4 import BeautifulSoup

from selenium import webdriver

driver = webdriver.Firefox()

driver.get('http://news.ycombinator.com')

html = driver.page_source

soup = BeautifulSoup(html)

for tag in soup.find_all('title'):
    print(tag.text)
    
Hacker News