Question

I want to visit every URL listed on a text file.

I have:

require 'mechanize'

File.open('sources.txt').each_line { |url|
  agent = Mechanize.new
  puts "Visiting #{url}"
  mypage = agent.get("#{url}")
  current_url = mypage.uri.to_s
  puts "The final URL is #{current_url}"
}

This code runs through the URLs in sources.txt, tells you it’s doing so, then tells you the URL once it reaches the page. i.e. if www.foo.com is in the source.txt, but www.foo.com resolves to just foo.com (no WWW), it will tell you so.

How can I save EACH resolved/final/current URL into a single, line separated text file?

Answer 1

If you want to append to a file, you use ‘a’ flag instead of ‘w’. Flags are here http://www.ruby-doc.org/core-1.9.3/IO.html

50.times do |n|
  File.open 'destinations.txt', 'a' do |file|
    file.write "line #{n}\n"
  end
end

Update (some refactoring to your code):

require 'mechanize'
agent = Mechanize.new

output = File.open 'output.txt', 'a'
File.open('sources.txt').each_line do |url|
  puts "Visiting #{url}"
  mypage = agent.get("#{url}")
  @current_url = mypage.uri.to_s
  puts "The final URL is #{@current_url}"
  output.puts @current_url
end
output.close

This way you will open each file once.