Question

I wrote some basic Python code to scrape a remote webpage and grab a few pieces of data. On a different page I’m trying to scrape, the data is hidden from view, and only appears after changing the value of a <select> box.

After de-minifying and digging through the remote website’s javascript, I confirmed that it is using AJAX (custom implementation of Prototype I think) to switch the <tbody> of the <table> I’m interested in scraping.

Is there a way to use Python (or Javascript via Python) to trigger the onChange event of that select box so I can “refresh” the DOM and grab the new HTML?

Answer 1

Figure out the AJAX request URL and request it directly. 🙂