Question

I want to update an erb.html page in rails with information from a webscraping script I wrote, but I don’t want the page to have to refresh every time new info comes in.

I don’t know where to put the background script, how rails will invoke it, etc.

How do I do this?

Answer 1

There are a number of ways to achieve your desired result in terms of construction but in the end you are going to have use Javascript and an AJAX call as there is no other way to update the content on your page with the scraped data.

Here’s how I use AJAX and screen-scraping for RoR:

application.js

In our application JS file we listen on a field for when it is changed (the user inputs a web url or some kind of data) and then we make an AJAX call to our scraper (url set up in Routes.rb later) and our data is returned in JSON format (when the success trigger of AJAX is called) which we loop through and display the scraped data in two fields that are named in a similar format as the key of our JSON key, value pairs.

$('#item_url').live('change', function() {          
     $.ajax({
            url: "/scrape/siteandnum", 
            data: {item_url: document.getElementById('item_url').value},             
            dataType: 'json', 
            success: function(data) {                                                               
                    $.each(data, function(key, value) {                        
                        document.forms['new_qued_item'].elements['qued_item_' + key].value = value
                    });         
                }
            });
}); 

qued_items_controller.rb

In our controller we have a method which is the Ruby scrape code. It takes in the params of the item_url (passed in from the data of the AJAX call) — in this example it takes in an eBay auction URL and gives us the eBay site domain and the item number.

def scrape_site_and_num
    url = params[:item_url] 
    itemnumber = url.match(/\d{12}/)[0].to_i 
    site = url.match(/ebay.(co.uk|com|de|fr|it|ca)/)[0]

    render :json => {'itemnumber' => itemnumber, 'site' => site}
end 

routes.rb

Of course we have to have a route so that the AJAX call knows where to access our scrape method.

  get "scrape/siteandnum/" => "qued_items#scrape_site_and_num"

So there you have it. Using this method any kind of data can be entered and a “behind-the-scenes” Ruby method will process the user input and come back with some screen-scraped data without the page ever reloading!