Question

I am using URI.unescape to unescape text, unfortunately I run into weird error:

 # encoding: utf-8
 require('uri')
 URI.unescape("%C3%9Fą")

results in

 C:/Ruby193/lib/ruby/1.9.1/uri/common.rb:331:in `gsub': incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError)
    from C:/Ruby193/lib/ruby/1.9.1/uri/common.rb:331:in `unescape'
    from C:/Ruby193/lib/ruby/1.9.1/uri/common.rb:649:in `unescape'
    from exe/fail.rb:3:in `<main>'

why?

Answer 1

The implementation of URI.unescape is broken for non-ASCII inputs. The 1.9.3 version looks like this:

def unescape(str, escaped = @regexp[:ESCAPED])
  str.gsub(escaped) { [$&[1, 2].hex].pack('C') }.force_encoding(str.encoding)
end

The regex in use is /%[a-fA-F\d]{2}/. So it goes through the string looking for a percent sign followed by two hex digits; in the block $& will be the matched text (‘%C3’ for example) and $&[1,2] be the matched text without the leading percent sign ('C3'). Then we call String#hex to convert that hexadecimal number to a Fixnum (195) and wrap it in an Array ([195]) so that we can use Array#pack to do the byte mangling for us. The problem is that pack gives us a single binary byte:

> puts [195].pack('C').encoding
ASCII-8BIT

The ASCII-8BIT encoding is also known as “binary” (i.e. plain bytes with no particular encoding). Then the block returns that byte and String#gsub tries to insert into the UTF-8 encoded copy of str that gsub is working on and you get your error:

incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError)

because you can’t (in general) just stuff binary bytes into a UTF-8 string; you can often get away with it:

URI.unescape("%C3%9F")         # Works
URI.unescape("%C3µ")           # Fails
URI.unescape("µ")              # Works, but nothing to gsub here
URI.unescape("%C3%9Fµ")        # Fails
URI.unescape("%C3%9Fpancakes") # Works

Things start falling apart once you start mixing non-ASCII data into your URL encoded string.

One simple fix is to switch the string to binary before try to decode it:

def unescape(str, escaped = @regexp[:ESCAPED])
  encoding = str.encoding
  str = str.dup.force_encoding('binary')
  str.gsub(escaped) { [$&[1, 2].hex].pack('C') }.force_encoding(encoding)
end

Another option is to push the force_encoding into the block:

def unescape(str, escaped = @regexp[:ESCAPED])
  str.gsub(escaped) { [$&[1, 2].hex].pack('C').force_encoding(encoding) }
end

I’m not sure why the gsub fails in some cases but succeeds in others.