Question

I am making a small scraper for the yahoo.finance.com website. When I make this request:

symbol = 'AAPL'

@page = Nokogiri::HTML(open("http://finance.yahoo.com/q?s=#{symbol.upcase}&ql=1"))

def marketCap(symbol)
  @page.xpath("//*[(@id = \"yfs_j10_#{symbol.downcase}\")]").text
end

puts marketCap(symbol)

It prints two times the same result.

“495.74B495.74B”

I looked at the source code and the tag is only showing it once

<span id="yfs_j10_f">51.74B</span>

If I use a css selector instead I get the same problem.

Is it a bug or did I made a mistake?

Thanks.

Answer 1

@page.xpath("//*[(@id = \"yfs_j10_#{symbol.downcase}\")]").text

isn’t correct.

xpath returns a NodeSet, which is similar to an Array. If it contains two elements text will contain both of them:

@page.xpath("//*[(@id = \"yfs_j10_#{symbol.downcase}\")]").size
=>2

Instead, use at_xpath to find the first one.

@page.at_xpath("//*[(@id = \"yfs_j10_#{symbol.downcase}\")]").text
=> "495.74B"

Now, instead of using XPath, which I feel is usually more complicated and less readable, I’d recommend using CSS for your accessor:

@page.at("#yfs_j10_#{symbol.downcase}").text
=> "495.74B"

Notice that I used at instead of at_css or at_xpath. at senses whether you’re passing an XPath or CSS. It’s generic, and could make a mistake figuring out which to use, but it’s also easier to use. The same is true of search instead of css or xpath. It returns a NodeSet like the other two, but senses which type of accessor it should use.