Question

I found some solutions using post_connect_hook and pre_connect_hook, but it seems like they don’t work. I’m using the latest Mechanize version (2.1). There are no [:response] fields in the new version, and I don’t know where to get them in the new version.

• https://gist.github.com/search?q=pre_connect_hooks
• https://gist.github.com/search?q=post_connect_hooks

Is it possible to make Mechanize return a UTF8 encoded version, instead of having to convert it manually using iconv?

Answer 1

Since Mechanize 2.0, arguments of pre_connect_hooks() and post_connect_hooks() were changed.

See the Mechanize documentation:

pre_connect_hooks()

A list of hooks to call before retrieving a response. Hooks are called with the agent, the URI, the response, and the response body.

　

post_connect_hooks()

A list of hooks to call after retrieving a response. Hooks are called with the agent, the URI, the response, and the response body.

Now you can’t change the internal response-body value because an argument is not array. So, the next best way is to replace an internal parser with your own:

class MyParser
  def self.parse(thing, url = nil, encoding = nil, options = Nokogiri::XML::ParseOptions::DEFAULT_HTML, &block)
    # insert your conversion code here. For example:
    # thing = NKF.nkf("-wm0X", thing).sub(/Shift_JIS/,"utf-8") # you need to rewrite content charset if it exists.
    Nokogiri::HTML::Document.parse(thing, url, encoding, options, &block)
  end
end

agent = Mechanize.new
agent.html_parser = MyParser
page = agent.get('http://somewhere.com/')
...