Question

I’m parsing through a website and i’m looking for potentially many million rows of content. However, csv/excel/ods doesn’t allow for more than a million rows.

That is why I’m trying to use a provisionary to exclude saving empty content. However, it’s not working: My code keeps creating empty rows in csv.

This is the code I have:

# create csv
CSV.open("neverending.csv", "w") do |csv|
csv << ["kuk","date","name"]

# loop through all urls
File.foreach("neverendingurls.txt") do |line|
begin
doorzoekbarefile = Nokogiri::HTML(open(line))
for k in 1..999 do

# PROVISIONARY / CONDITIONAL
unless doorzoekbarefile.at_xpath("//td[contains(style, '60px')])[#{k}]").nil?

# xpaths
kuk = doorzoekbarefile.at_xpath("(//td[contains(@style,'60px')])[#{k}]")
date = doorzoekbarefile.at_xpath("(//td[contains(@style, '60px')])[#{k}]/following-sibling::*[1]")
name = doorzoekbarefile.at_xpath("(//td[contains(@style, '60px')])[#{k}]/following-sibling::*[2]")

# save to csv
csv << [kuk,date,name]

end
end
end

rescue
puts "error bij url #{line}"
end
end
end

Anybody have a clue what’s going wrong or how to solve the problem? Basically I simply need to change the code so that it doesn’t create a new row of csv data when the xpaths are empty.

Answer 1

This really doesn’t have to do with xpath. It’s simple Array#empty?

row = [kuk,date,name]
csv << row if row.compact.empty?

BTW, your code is a mess. Learn how to indent at least beore posting again.