Question

I am scraping a webpage for text, using the code as follows

doc.xpath("//td[text()='Operating system']/following-sibling::td")
doc.xpath("//td[text()='Processors']/following-sibling::td")

I have about 30 of these so I thought I could maybe use an array but its not working, here is my code

clues = Array.new
clues << 'Operating system'
clues << 'Processors'
clues << 'Chipset'

clues.each do |clue_storeage|
doc.xpath("//td[text()=#{clues}]/following-sibling::td")
end

Is there a way I can feed the array into that loop and then output that to a CSV?

Answer 1

To clarify mb2nd’s comment, your each block is incorrectly referencing the array. This should work:

clues.each do |clue|
  doc.xpath("//td[text()=#{clue}]/following-sibling::td")
end

To output the captured data to CSV you could run:

csv = ""
clues.each do |clue|
  csv << doc.xpath("//td[text()=#{clue}]/following-sibling::td")
  csv << ", " unless clues.last == clue
end

The doc.xpath(“//td[text()=#{clue}]/following-sibling::td”) call may need .value on the end?

On a side note; you may also populate your array like so:

clues = ['Operating system', 'Processors', 'Chipset']  

---

EDIT (after last comment from @Ninja2K)

You need to save the result of each the xpath calls. Here’s some working code:

require 'rubygems' 
require 'nokogiri' 
require 'open-uri' 

doc = Nokogiri::HTML(open("http://h10010.www1.hp.com/wwpc/ie/en/ho/WF06b/321957-321957-3329742-89318-89318-5186820-5231694.html?dnr=1%22"))

clues = ['Operating system', 'Processors', 'Chipset'] 

csv_text = ""
clues.each do |clue|
  csv_text << doc.at_xpath("//td[text()='#{clue}']/following-sibling::td").text
  csv_text << ", " unless clues.last == clue
end
puts csv_text

BTW. You may also find this article useful: http://hunterpowers.com/data-scraping-and-more-with-ruby-nokogiri-sinatra-and-heroku/