Does anyone ever see a lot of errors like this:
Exception `Net::HTTPBadResponse' at /usr/lib/ruby/1.8/net/http.rb:2022
- wrong status line: _SOME HTML CODE HERE_
When using threads and mechanize? I’m relatively certain that this is
some bad behavior between threads and the net/http library, but does
anyone have any advice as far as the upper limit of threads you want
to run at once when using mechanize/nethttp? And how can I capture this kind of exception because rescue Net::HTTPBadResponse doesn’t work?
This could be something non-thread-safe in Mechanize, but I can think of other bugs that might cause the same problem. I’d start by disabling persistent connections, if you’re using them. The next thing to do is to look at your code, and make sure that you’re being careful with the objects you handle. If your application has multiple threads mucking about with common objects, that can break a library that would be otherwise thread-safe.
If there is a threading problem somewhere, the upper limit of threads you can use safely is 1. Any more, and you’re just making a trade-off about how often you want the problem to occur, rather than whether it occurs or not.