Related question: Set session to scrape page
As a personal project, I’m trying to scrape my University’s course catalog to provide it as a RESTful API.
URL1: https://duapp3.drexel.edu/webtms_du/
URL2: https://duapp3.drexel.edu/webtms_du/Colleges.asp?Term=201125&univ=DREX
Going to URL1 sets a cookie which is needed for URL2. I’m successfully receiving and sending the cookie like this:
curl -c recd url1
curl -v -b recd url2
However, the response I get differs from the one I see in my browser. Basically, my browser shows me a list of majors, while curl gets a page where instead of the list of majors, it has
<FONT COLOR=red size=5> The lists of subjects are not available online for the selected term at this time.</FONT><FONT COLOR=red size=5> Please check back again later.</FONT>
I figured maybe they were checking user-agent, so I tried this as well:
curl -b recdcookies --user-agent "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)" https://duapp3.drexel.edu/webtms_du/Colleges.asp?Term=201125&univ=DREX
But same unexpected response.
What is causing the different outputs between browser and curl? As far as curl -v shows it only sets one cookie. Is it because they are https pages, and I need to be adding another argument to the curl ?
When using curl on the command line, you must enclose the url in quotes if it contains an ampersand (
&) because it has special meaning on unix systems.