I have been practicing my scraping abilities in Python. I have gotten pretty good, but came across a few sites that have me pretty stumped. They make use of Ajax to find the nearby locations. There are several sites designed the same way. One of the websites is http://www.applebees.com. Even using firebug I cannot find the answer.
How can Python request the locations via the ajax call? I am completely stumped.
The page is http://www.applebees.com, there is a form on the right hand side to enter the zipcode and it pulls up the closest locations to that zipcode. However, if I pull the source after this zipcode is entered the locations still don’t show up in the source file. The request/response are completely ajax and hidden to the html source, i have never seen anything like it. I am trying to research a solution now.
Scraping programatically using an http library can be difficult for some sites. If you are trying to simulate user interraction on a JavaScript heavy site (ajax or otherwise) you might consider driving a real browser using something like selenium. There are python client browsing and you will get some access to the page DOM.
http://pypi.python.org/pypi/selenium