I’m designing a link scraping program that grabs the basic link preview fields for a given URL, like page title, description, and images, etc. So far I’ve got a pretty good working version that uses the Python requests library and Beautiful Soup.
Most URLs come across perfectly, but when I try the url of a Facebook app, I get a different HTML response than if I accessed it from a browser directly. For instance, if I navigate to the app in a browser and view-source, I’ll see a title field specific to that app. However, the HTML response in Python returns the generic Facebook.com title field.
I’m trying to understand how it is that Facebook app page is delivering a certain HTML response to my browser, and another one to my Python server.
Facebook app example:
http://www.facebook.com/cocacola/app_106795496113635
From browser response:
<title>Coca-Cola</title>
From Python ‘requests’ response:
<title>Facebook</title>
Python code:
import requests
r = requests.get(url, allow_redirects=True)
html = r.text
print html
UPDATE:
OK, so just realized the Python response is for a Facebook login page. This is a public app though, so the question is why does it want to require login from my server.
Much easier is to use the chrome developer tools (Shift-Control-J or View->Developer->Developer Tools. Then go to the network tab, press the record button (a black circle by default when not recording, can be difficult to find at first). Then access facebook, highlight your request of choice, view the headers for that request in the sub-tabs. You’re likely looking for something like