Question

I would like to web-scrape the html source code of java-script pages that I can´t access without selecting one option in a drop-down list and, after, ‘clicking’ on links. Spite of not been in java, a simple example can be this:

Web-scrape the main wikipedia pages in all languages available in the drop-down list in the bottom of this url: http://www.wikipedia.org/

To do so, I need to select one language, English for example, and then ‘click’ in the ‘Main Page’ link in the left of the new url (http://en.wikipedia.org/wiki/Special:Search?search=&go=Go).

After this step, I would scrape the html source code of the wikipedia main page in English.

Is there any way to do this using R? I have already tried RCurl and XML packages, but it does not work well with the javascript page.

If it is not possible with R, could anyone tell me how to do this with python?

Answer 1

It’s possible to do this using python with the selenium package. There are some useful examples here. I found it helpful to install Firebug so that I could identify elements on the page. There is also a Selenium Firefox plugin with an interactive window that can help too.

import sys
import selenium
from selenium import webdriver
from selenium.webdriver.common.keys import Keys   

driver = webdriver.Firefox()
driver.get("http://website.aspx")    

elem = driver.find_element_by_id("ctl00_ctl00")
elem.send_keys( '15' )
elem.send_keys( Keys.RETURN )