Question

I am using mechanize to scrap some web pages.

• I need to know what are mechanize limitations? What mechanize can not do?
• Can it execute javascripts embedded in the web page?
• Can I use it to call javascript functions? I don’t think it can. I think Watir can.
• What are the differences between it and watir?

Answer 1

Mechanize can do a lot. It uses net/http so whatever you can do with net/http you can do with mechanize. Although it supports much more as per their description :

The Mechanize library is used for automating interaction with websites. Mechanize automatically stores and sends cookies, follows redirects, can follow links, and submit forms. Form fields can be populated and submitted. Mechanize also keeps track of the sites that you have visited as a history.

Check out this link for some information on using javascript with mechanize: here

It would be much easier to tell you if mechanize support a specific function/task instead of going through everything. What are you looking to do exactly ?

Javascript is the one thing mechanize can’t do. The one thing it does support most of the time is displaying Javascript links. ie using page.links.each {|link| puts link.text} will also display Javascript, but you won’t be able to click/select them.

In simple terms Watir does support Javascript. It’s actually your browser that supports javascript and Watir controls the browser.

Watir runs a real browser(FF,Chrome,IE) and programmatically controls that browser. It acts exactly like a user would when accessing a website. This is what enables you to use javascript. Watir only controls the browser and the browser is the one sending requests and getting responses and rendering/processing it all. You are limited by the speed of the browser you use.

Mechanize on the other hand acts like its own ‘browser’ and is much faster than Watir, becomes it does not render pages. It talks directly with the server, and processes the raw response. Mechanize is limited by your connection speed.

Watir would be used over Mechanize when you need to watch and see what’s happening, use javascript, or do anything GUI related. Mechanize is much faster and is good for testing the actual structure of the website. (testing links/logins/etc)