Question

I am trying to get youtube links embedded in a webpage. When I retrieve the page using urllib2 I don’t see the link in the response. I think the embedded video is retrieved by a script on the page which gets loaded when I use a browser. How do I get the same using python’s urllib2 ?

A sample webpage is http://busymovies.appspot.com/News.html?id=2965032

Answer 1

To extract dynamically generated (with javascript) content you could use selenium:

#!/usr/bin/env python
from contextlib import closing
from selenium.webdriver import Firefox # pip install selenium

url = "http://busymovies.appspot.com/News.html?id=2965032"

# use firefox to get page with javascript generated content
with closing(Firefox()) as browser:
    browser.get(url)
    link = browser.find_element_by_link_text("Direct Link")
    print link.get_attribute("href")

Output

http://www.youtube.com/v/nLJYkat4HpE&hl=en_US&feature=player_embedded&version=3