Sign Up to our social questions and Answers Engine to ask questions, answer people’s questions, and connect with other people.
Login to our social questions & Answers Engine to ask questions answer people’s questions & connect with other people.
Lost your password? Please enter your email address. You will receive a link and will create a new password via email.
Please briefly explain why you feel this question should be reported.
Please briefly explain why you feel this answer should be reported.
Please briefly explain why you feel this user should be reported.
the html that I am receiving from urllib2 is missing dozens of fields of data that I can see when I view the source of the URL in Firefox. Any advice would be much appreciated. Here is what it looks like:
from FireFox view source:
# ...<td class=td6>as</td></tr></thead>|ManyFields|<br></div><div id="c1">...
from urllib2 return html:
# ...<td class=td6>as</td></tr></thead>|</table>|<br></div><div id="c1">...
It seems from a cursory check that the page you’re getting has a lot of Javascript; perhaps that Javascript cooperates in building the information that you see at the end in Firefox (at least some of it is actively altering the page’s contents). If you need to scrape JS-rich pages, your best bet is to automate an actual browser via Selenium.