Question

I’m testing a module that does webscraping.

I’ve got this method that grabs the robots.txt file for a page.

def download_robots_file(page)
  Net::HTTP.get(URI("#{page}robots.txt"))
rescue StandardError => ex
  log_info('robot_file', ex)
end

And I’ve got this tests. The first spec ensures that the method can fail safely and the second test covers what happens when there is actually a page to be downloaded. What I’ve done is to download the robots.txt file fromm cnn.com and store it locally. I copied the file and chopped off the last bit (“robots.txt”) so that my method acts works normally. A bit strange, I admit. I’m open to a better way to do things.

describe '#download_robots_file(page)' do
  it "returns if there's no page" do
    @dummy.stub(:log_info).and_return("No Robots.txt file exists.")
    page = ''
    @dummy.download_robots_file(page).should == "No Robots.txt file exists." 
  end

  it "returns the robots file if it exists" do
    page = './spec/data/cnn_' #gotta be better way!
    robots_file = File.open('./spec/data/cnn_robots.txt', "r")
    expected_page = robots_file.read
    @dummy.stub(:log_info)
    @dummy.download_robots_file(page).should == expected_page
  end
end

My questions are:

1. Is this a good strategy for testing whether the download_robots_file method is working properly?
2. If not, what’s a better way to do this?
3. Is there a better way to than using the “.and_return” code in the first test?

Answer 1

Another option is to use webmock.

https://github.com/bblimke/webmock/

Once you look at either webmock or fakeweb all your 3 questions will be answered