Question

I need to build with ruby the following steps:
1. enter the homepage of a website
2. in there is a specific html form I need to enter user\pass in it and click the input button (i.e. html post)
3. get the response (html + login cookie) from step 3.
4. do many html get request with the login cookie (otherwise I won’t have permission for the page)

BTW I tried

url = URI.parse('http://www.example.com/todo.cgi')
req = Net::HTTP::Post.new(url.path)
req.basic_auth 'jack', 'pass'
req.set_form_data({'from'=>'2005-01-01', 'to'=>'2005-03-31'}, ';')
res = Net::HTTP.new(url.host, url.port).start {|http| http.request(req) }
case res
when Net::HTTPSuccess, Net::HTTPRedirection
  # OK
else
  res.error!
end

and it didn’t work.

Thank you guys

Answer 1

There are several excellent Ruby gems for automated access of web sites.

See:

• the Ruby Toolbox for HTML Parsing
• the Ruby Toolbox for HTTP clients

Nokogiri is the most popular gem for HTML parsing. Ryan Bates has an excellent screencast on Mechanize. Mechanize is a Ruby library for automated interaction with websites. It extends Nokogiri allowing you to interact with multiple pages on the site: click links, submit forms, etc.

Take a look at Mechanize. You’ll get far with a well-documented Ruby library and save the work of implementing it yourself.