Question

I want to be able to do something like this in Ruby:

(new[]{"http://google.com", "http://facebook.com", "http://microsoft.com", "http://twitter.com", "http://stackoverflow.com"})
.AsParallel()
.Select(q => WebRequest.Create(q).GetResponse())

This also should preserve order of elements and use deferred execution.
Are there any options available?

Answer 1

The Ruby curb gem, which is a wrapper around the curl library, has a “multi” mode that can fetch a number of URLs in parallel.

From the example:

# Make multiple GET requests
easy_options = { :follow_location => true }
multi_options = { :pipeline => true }

Curl::Multi.get('url1','url2','url3','url4','url5', easy_options, multi_options) do |easy|
  # do something interesting with the easy response
  puts easy.last_effective_url
end