I’m trying to debug an issue that’s started coming up recently in my Rails (3.0.9) app, in which instances of the app have to be restarted (not redeployed) when a non-breaking migration is performed on the database. This did not used to happen, and I can’t find a reason it should have started recently.
My production environment has a few servers, each of which is using forked Unicorn instances, all sharing a single MySQL instance. Each instance is using a connection pool of 1. I also have one server that’s used as a final staging environment to test against live data, after testing against a staging database.
In the past, we’ve been able to run migrations on the production DB without redeploying or even restarting the production Unicorn servers as long as they were migrations that the old code never needed to see. For example, adding a new, optional field to an existing table — the code deployed to the staging server would write data to it, but the existing servers could blissfully ignore it and continue to use the DB the same way it had been until we’re ready to push the new code to production.
Recently, though, we’ve been seeing an exception being thrown by production servers after running the migration, but only when it tries to insert a new row into the modified table. All other parts of the app, and all other tables, are unaffected, but as soon as there’s an insert, we get this:
NoMethodError: undefined method `name' for nil:NilClass
[GEM_ROOT]/gems/arel-2.0.10/lib/arel/visitors/to_sql.rb:56:in `visit_Arel_Nodes_InsertStatement'
[GEM_ROOT]/gems/arel-2.0.10/lib/arel/visitors/to_sql.rb:55:in `map'
[GEM_ROOT]/gems/arel-2.0.10/lib/arel/visitors/to_sql.rb:55:in `visit_Arel_Nodes_InsertStatement'
[GEM_ROOT]/gems/arel-2.0.10/lib/arel/visitors/visitor.rb:15:in `send'
[GEM_ROOT]/gems/arel-2.0.10/lib/arel/visitors/visitor.rb:15:in `visit'
[GEM_ROOT]/gems/arel-2.0.10/lib/arel/visitors/visitor.rb:5:in `accept'
[GEM_ROOT]/gems/arel-2.0.10/lib/arel/visitors/to_sql.rb:18:in `accept'
[GEM_ROOT]/bundler/gems/rails-83fb5552b6ab/activerecord/lib/active_record/connection_adapters/abstract/connection_pool.rb:111:in `with_connection'
[GEM_ROOT]/gems/arel-2.0.10/lib/arel/visitors/to_sql.rb:16:in `accept'
[GEM_ROOT]/gems/arel-2.0.10/lib/arel/tree_manager.rb:20:in `to_sql'
[GEM_ROOT]/gems/arel-2.0.10/lib/arel/select_manager.rb:217:in `insert'
The issue goes away if we then restart the production servers, even though they’re still running the old code. So it seems that something gets corrupted in the connection when the table is modified, even though the app, if it hadn’t hit this error, would be fine continuing to do what is was doing before the migration.
I did some digging into ActiveRecord/ARel, but the most I can do is theorize that at some point the cached model of this table loses any knoweledge of which columns it has, but I can’t see why it would do that or why this would be happening all of a sudden.
I’m a coworker of Jon’s, going to answer this for posterity.
The problem was a consequence of the caching in https://github.com/rails/rails/blob/v3.0.9/activerecord/lib/active_record/base.rb; essentially @columns and @arel_table for certain model classes were out of sync because the latter was cached during app initialization and thus inherited from the master Unicorn by any new worker.
We fixed it by preventing calls to any model’s ::scoped or ::unscoped during initialization where we could, and calling ::reset_column_information afterwards where we couldn’t.