Question

I’m new to YQL, and just trying to learn how to do some fairly simple tasks.

Let’s say I have a list of URLs and I want to get their HTML source as a string in javascript (so I can later insert it to a database via ajax). How would I go about getting this info back in Javascript? Or would I have to do it in PHP? I’m fine with either, really – whatever can work.

Here’s the example queries I’d run on their console:

select * from html where url=”http://en.wikipedia.org/wiki/Baroque_music”

And the goal is to essentially save the HTML or maybe just the text or something, as a string.

How would I go about doing this? I somewhat understand how the querying works, but not really how to integrate with javascript and/or php (say I have a list of URLs and I want to loop through them, getting the html at each one and saving it somewhere).

Thanks.

Answer 1

You can’t read other pages with Javascript due to a built-in security feature in web browsers. It is called the Same origin policy.

The usual method is to scrape the content of these sites from the server using PHP.

There is an other option with javascript called a bookmarklet.

You can add the bookmarklet in your bookmarks bar, and each time you want the content of a site click the bookmark.

A script will be loaded in the host page, it can read the content and post it back to your server.

Oddly enough, the same origin policy, does not prevent you to POST data from this host page to your domain. You need to POST a FORM to an IFRAME that has a source hosted on your domain.

You won’t be able to read the response you get back from the POST.
But you can poll with a setInterval making a JSONP call to your domain to know if the POST was successful.