Question

How do I translate or strip character sequences like “\xC2\xBB” in my strings in Ruby 1.9.2?

Answer 1

You will usually see hex bytes like that when the string is using an encoding that does not handle those bytes. If you know what encoding the string is supposed to be using, you can use String#force_encoding to re-interpret the bytes according to your desired encoding.

# Under a UTF-8 locale:
ruby-1.9.2-head :013 > "\xC2\xBB".force_encoding(Encoding::UTF_8)
 => "»" 

# Under the “C” locale:
ruby-1.9.2-head :007 > "\xC2\xBB".force_encoding(Encoding::UTF_8)
 => "\u00BB" 

Both result in the same UTF-8 encoded string internally. When under the C locale, Ruby prints an escaped version to avoid printing binary data to the terminal (which, according to the locale setting, might not support it).

If the string is already using the appropriate encoding, then you should re-encode the string to your desired output encoding before using it:

# Under a UTF-8 locale:
ruby-1.9.2-head :026 > "\xC2\xBB".force_encoding(Encoding::ISO_8859_1).encode(Encoding::UTF_8)
 => "»" 

# Under the “C” locale:
ruby-1.9.2-head :014 > "\xC2\xBB".force_encoding(Encoding::ISO_8859_1).encode(Encoding::UTF_8)
 => "\u00C2\u00BB" 

Above, I use String#force_encoding to make sure the bytes in the string are are flagged as ISO 8859-1 (because, for instance, a header accompanying the bytes said that they represented an ISO 8859-1 encoded string) and then use String#encode re-encode it as UTF-8 (the desired output encoding).

Finally, if you really just want to strip out anything that is not ASCII, you could use the negated [:ascii:] character class with String#gsub:

ruby-1.9.2-head :030 > "foo\xC2\xBBbar".force_encoding(Encoding::UTF_8).gsub(/[[:^ascii:]]/,'')
 => "foobar"