Question

I’ve got a Rails 3.2.8 project with both legacy test/unit (minitest) tests and RSpecs. Although we have Jenkins running continuous integration, we haven’t noticed failures in a while. The reason is that something seems to be overriding the exit code on a test failure and returning an exit code of 0 regardless whether there is a failing test, or spec.

I’ve done some digging, but would like to dig into the Kernel#at_exit stack to see what gems/plugins are registering a proc. I know it’s a language feature that’s implemented in C.

So, I’m looking for help shaving this yak:

• Why do my failing tests return exit code of 0?
• Assuming something is adding a proc/callback to at_exit, how to I get in to see what the callbacks are?
  • I know it’s written in C, and have looked at the source. I don’t see any ruby hooks into it. Are there 3rd party extensions that can be added, or is anyone aware of a patch to ruby that allows some visibility there?
  • Would dtrace work to get in there and look?

Some things I’ve done:

• I’ve grepped the gems and the installed libraries for at_exit but didn’t find anything too suspicious.
• Using the same gemset in my Rails app, if I create a new app with a single spec and run it it works as expected (passing tests return exit code 0 and failures return 1)
• Looked to find patches/extensions to ruby that help provide visibility into at_exit

Some things I haven’t done yet:

• Tried to learn DTrace (I’m on a mac) – I’ve seen presentations showing how to use it, but I’ve not tried yet.
• remove every gem from my project to see what might be causing it — it does seem to be project specific.

Thanks for your help in advance!

P.S.: Here are the gems in this project’s Gemfile:

gem 'rails', '~> 3.2.8'
gem 'acts-as-taggable-on', '~> 2.3.1'
gem "akamai_api", :path => "vendor/git/akamai_api"
gem 'dalli'
gem "gabba"
gem "googlecharts", :require => "gchart"
gem "hpricot"
gem 'jquery-rails'
gem 'linkscape'
gem "logging", "1.4.3"
gem 'machinist', '>= 2.0.0.beta2'
gem "mysql2"
gem 'newrelic_rpm'
gem "premailer-rails3"
gem 'rack-cache'
gem "rack-ssl-enforcer"
gem 'sendgrid'
gem "tokyocabinet", "1.29"
gem 'twitter-bootstrap-rails', :git => 'git://github.com/seyhunak/twitter-bootstrap-rails.git'
gem "useragent"
gem "yajl-ruby", :require => "yajl"

group :assets do
  gem 'coffee-rails'
  gem 'sass-rails'
  gem 'uglifier'
end

group :no_rake do
  gem "vanity"
end

group :development, :test do
  gem 'debugger'
  gem 'thin'
  gem 'rspec-rails'
end

group :test do
  gem 'vanity'
  gem 'minitest-rails'
  gem 'mocha'
end

As an update, I did a binary tree search in the Gemfile (turn half off, run the tests, turn the other half off.. repeat) until I got the tests to fail properly. The logging gem was causing rspec to return nonzero, and the googlecharts gem was messing up test/unit. At this point, I don’t know why.

I haven’t dug in further to see which behavior(s) of the libraries (or the dependent code in my app) are affecting the tests. A quick grep didn’t return anything using Kernel#at_exit and a quick glance of the source on each of those didn’t turn up anything either. I’ll keep digging, but that’s what I found out so far.

I haven’t filed bug reports, as I don’t yet know what the problem is.

Answer 1

You gotta love ruby 🙂

alias original_at_exit at_exit
def at_exit(*args, &block)
  puts "registered at exit: #{caller * "\n"}"
  original_at_exit *args, &block
end