Further to the title, when loading data from ActiveRecord the encoding is always set to ASCII-8Bit in spite of my best efforts to force the encoding. I have entered as much detail as possible here to try and build a good error report someone could use to help me out!
The project is using the following technologies:
- Padrino Framework
- Ruby 1.9.2-rc2 (Also 1.9.1 and 1.9.2-preview3)
- ActiveRecord
- MySQL
(Full List)
$ bundle show | ack '(record|padrino)'
* activerecord (2.3.8)
* padrino (0.9.14)
* padrino-admin (0.9.14)
* padrino-core (0.9.14)
* padrino-gen (0.9.14)
* padrino-helpers (0.9.14)
* padrino-mailer (0.9.14)
Episodes Table:
mysql> DESCRIBE `episodes`;
+----------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+----------------+--------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| show_id | int(11) | YES | | NULL | |
| season_id | int(11) | YES | | NULL | |
| episode_number | int(11) | YES | | NULL | |
| title | varchar(255) | YES | | NULL | |
| year | int(11) | YES | | NULL | |
+----------------+--------------+------+-----+---------+----------------+
6 rows in set (0.02 sec)
mysql> SHOW CREATE TABLE episodes;
Table: episodes
Create Table: CREATE TABLE `episodes` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`show_id` int(11) DEFAULT NULL,
`season_id` int(11) DEFAULT NULL,
`episode_number` int(11) DEFAULT NULL,
`title` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`year` int(11) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=74332 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
mysql> SHOW CREATE DATABASE development;
+-------------+--------------------------------------------------------------------------------------------------------+
| Database | Create Database |
+-------------+--------------------------------------------------------------------------------------------------------+
| development | CREATE DATABASE `development` /*!40100 DEFAULT CHARACTER SET utf8 COLLATE utf8_unicode_ci */ |
+-------------+--------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)
As you can see the database certainly thinks that things should be UTF-8; and the database adapter too:
ActiveRecord::Base.configurations[:development] = {
:adapter => 'mysql',
:encoding => 'utf8',
:reconnect => false,
:database => "development",
:pool => 5,
:username => 'root',
:password => '',
:host => 'localhost',
}
That is echoed in the console when examining the active record connection:
ruby-1.9.2-rc1 > ActiveRecord::Base.connection
DEBUG - [06/Jul/2010 19:24:32] "SQL (0.1ms) SET NAMES 'utf8'"
DEBUG - [06/Jul/2010 19:24:32] "SQL (0.1ms) SET SQL_AUTO_IS_NULL=0"
=> #<ActiveRecord::ConnectionAdapters::MysqlAdapter:0x0000010936fa88 @logger=#<Padrino::Logger:0x00000101587198 @buffer=[], @auto_flush=true, @level=0, @log=#<IO:<STDOUT>>, @mutex=#<Mutex:0x00000101587148>, @format_datetime="%d/%b/%Y %H:%M:%S", @format_message="%s - [%s] \"%s\"">, @connection=#<Mysql:0x0000010936fad8>, @runtime=0.2608299255371094, @last_verification=0, @query_cache_enabled=false, @config={:adapter=>"mysql", :encoding=>"utf8", :reconnect=>false, :database=>"development", :pool=>5, :username=>"root", :password=>"", :host=>"localhost"}, @connection_options=["localhost", "root", "", "development", nil, nil, 131072], @quoted_table_names={}, @quoted_column_names={}>
ruby-1.9.2-rc1 > ActiveRecord::Base.connection.encoding
Ruby should know the language, here’s my $ locale
LANG="en_GB.UTF-8"
LC_COLLATE="en_GB.utf-8"
LC_CTYPE="en_GB.utf-8"
LC_MESSAGES="en_GB.utf-8"
LC_MONETARY="en_GB.utf-8"
LC_NUMERIC="en_GB.utf-8"
LC_TIME="en_GB.utf-8"
LC_ALL=
Although Ruby is not setting Encoding.default_internal:
$ irb --simple-prompt
ruby-1.9.2-rc1 > Encoding.default_internal
=> nil
I have added a snippet in my application’s config/boot.rb that looks like this:
if Kernel.const_defined?("Encoding") and Encoding.respond_to?(:find) and Encoding.respond_to?(:default_internal)
Encoding.default_internal = Encoding.find('UTF-8')
end
That works exactly as you might expect… but is a hack, and doesn’t solve the problem.
And here’s the output of the problem in situ:
ruby-1.9.2-rc1 > e = Episode.new
DEBUG - [06/Jul/2010 19:29:14] "SQL (0.1ms) SET NAMES 'utf8'"
DEBUG - [06/Jul/2010 19:29:14] "SQL (0.1ms) SET SQL_AUTO_IS_NULL=0"
DEBUG - [06/Jul/2010 19:29:14] "Episode Columns (0.8ms) SHOW FIELDS FROM `episodes`"
=> #<Episode id: nil, show_id: nil, season_id: nil, episode_number: nil, title: nil, year: nil>
ruby-1.9.2-rc1 > e.title
=> nil
ruby-1.9.2-rc1 > nt = "New Title"
=> "New Title"
ruby-1.9.2-rc1 > nt.encoding
=> #<Encoding:UTF-8>
ruby-1.9.2-rc1 > e.title = nt
=> "New Title"
ruby-1.9.2-rc1 > e.title.encoding
=> #<Encoding:UTF-8>
ruby-1.9.2-rc1 > e.save
DEBUG - [06/Jul/2010 19:29:48] "SQL (0.1ms) BEGIN"
DEBUG - [06/Jul/2010 19:29:48] "Episode Create (0.2ms) INSERT INTO `episodes` (`show_id`, `season_id`, `episode_number`, `title`, `year`) VALUES(NULL, NULL, NULL, 'New Title', NULL)"
DEBUG - [06/Jul/2010 19:29:48] "SQL (0.4ms) COMMIT"
=> true
ruby-1.9.2-rc1 > Episode.find_by_title(nt).title.encoding
DEBUG - [06/Jul/2010 19:30:04] "Episode Load (29.5ms) SELECT * FROM `episodes` WHERE (`episodes`.`title` = 'New Title') LIMIT 1"
=> #<Encoding:ASCII-8BIT>
ruby-1.9.2-rc1 >
I had some success by overriding the accessors, and redefining them as:
class Episode
# ...
def title
title.encode!
end
# ...
end
Where encode! is defined here in the API docs for 1.9 – to quote it here “with no options returns a copy of str transcoded to Encoding.default_internal.”
Whilst my work-arounds are successful, I would much prefer to have UTF-8 coming out of the database, which is what my code seems to indicate to be the case.
You probably need the ruby-mysql gem, which is encoding aware in 1.9, instead of the more common mysql gem, which isn’t. See my blog for details.