Question

I have a web scraper built to parse html from a website and I’m trying to write tests for it.

The class I’m trying to test receives a Nokogiri HTML object and extracts the required data from it. Now as usual the html can vary, sometimes elements will be missing or whatnot. I need to test these different situations.

So what I’d like to do is make a bunch of html files, each one representing a case with a particular element missing etc. For each html file, I wish to also construct an associated hash of the data I would expect the scraper to extract, assuming it is working correctly.

So I would like to write a test which will iterate over these html files and compare the data extracted by the class being tested against the expected data and report whether or not it is correct.

Any suggestions as to how to do this?

Answer 1

Have a look at the Artifice, fakeweb or webmock gems, which override net/http in order to supply testable results.