Sign Up to our social questions and Answers Engine to ask questions, answer people’s questions, and connect with other people.
Login to our social questions & Answers Engine to ask questions answer people’s questions & connect with other people.
Lost your password? Please enter your email address. You will receive a link and will create a new password via email.
Please briefly explain why you feel this question should be reported.
Please briefly explain why you feel this answer should be reported.
Please briefly explain why you feel this user should be reported.
I am trying to use YQL to scrape some websites. When I test various queries in the YQL console I get an results node. So for example when I run:
select * from html where url="http://www.reverbnation.com/" and xpath='/html/body'
I get an empty <results /> node (permalink).
Thanks in advance!
http://www.reverbnation.com may be blocking the request coming from Yahoo! based on certain criteria, like headers. I had a look at reverbnation’s robots.txt, and they aren’t blocking Yahoo! based on the “Yahoo Pipes 2.0” user agent, so it must be something else.
To re-create the issue, make a YQL query against your own site, then look at the full access logs to see the full request and all headers that came from Yahoo! Then make a similar request using a tool like cURL.
You can also try and run netcat on a port and query with http://yoursite.com:PORT to see the full request.
Related issue discussed here.