Question

I am running into encoding problems:

db = SQLite3::Database.new "encoding.db"
=> #<SQLite3::Database:0x9b69cbc>

db.encoding
=> #<Encoding:UTF-16LE>

r1 = db.execute("select * from db_log_sink limit 2").first.last
=> "\u7953\u7473\u6D65\u5420\u6D69\u7A65\u6E6F\u3A65\u4520\u5453\u2F20\u4520\u5444\x0A"

r2 = db.execute("select * from db_log_sink limit 2").last.last
=> "????????\u0A3A"

r1.encoding
=> #<Encoding:UTF-16LE>

r2.encoding
=> #<Encoding:UTF-8>

Working at the linux command line with the same file I get:

select * from db_log_sink limit 2;
1|2011-11-16T12:02:15|0|System Timezone: EST / EDT
2|2011-11-16T12:02:15|0|Server Hostnames:

In the browser r2 comes out as Chinese traditional han. r1 comes out as normally formatted text.

About half the records in text columns come out garbled when using the gem/ruby. Everything looks normal at the Linux command line and using SQLiteSpy under Windows.

I have tried ~20 sqlite databases so far and they all show the same behaviour.

I can provide a download link for a db file if needed.

Any help would be much appreciated.

Answer 1

Found workaround:

r2.unpack('U*').pack('v*').force_encoding('utf-8')