Question

I’m currently struggling with percent escaping special characters on iOS, for instance “é” when contained in a query parameter value.

I’m using AFNetworking, but the issue isn’t specific to it.

The “é” character should be percent escaped to “%E9”, yet the result is “%C3%A9”. The reason is because “é” is represented as those 2 bytes in UTF8.

The actual percent escaping method is the well known one and I’m passing UTF8 as string encoding. The string itself is @”é”.

static NSString * AFPercentEscapedQueryStringPairMemberFromStringWithEncoding(NSString *string, NSStringEncoding encoding) 
{
    static NSString * const kAFCharactersToBeEscaped = @":/?&=;+!@#$()~";
    static NSString * const kAFCharactersToLeaveUnescaped = @"[].";

    return (__bridge_transfer  NSString *)CFURLCreateStringByAddingPercentEscapes(kCFAllocatorDefault, (__bridge CFStringRef)string, (__bridge CFStringRef)kAFCharactersToLeaveUnescaped, (__bridge CFStringRef)kAFCharactersToBeEscaped, CFStringConvertNSStringEncodingToEncoding(encoding));
}

I had hoped passing in UTF16 string encoding would solve it, but it doesn’t. The result is “%FF%FE%E9%00” in this case, it contains “%E9” but I must be missing something obvious.

Somehow I can’t get my head around it.
Any pointers would be awesome.

Answer 1

RFC 3986 explains that, unless the characters you’re encoding fall into the unreserved US-ASCII range, the convention is to convert the character to (in this case, A UTF8-encoded) byte value, and and use that value as the percent encoding base.

The behavior you’re seeing is correct.

The disparity between the encoded values given for UTF-8 vs. UTF-16 is due to a couple of factors.

Encoding Differences

First, there’s the difference in the way that the respective encodings are actually defined. UTF-16 will always use two bytes to represent its character, and essentially concatenates the higher order byte with the lower order byte to define the code. (The ordering of these bytes will depend on whether the code is encoded as Little Endian or Big Endian.) UTF-8, on the other hand, uses a dynamic number of bytes, depending on where in the Unicode code page the character exists. The way UTF-8 relates how many bytes it’s going to use is by the bits that are set in the first byte itself.

So if we look at C3 A9, that translates into the following bits:

1100 0011 1010 1001

Looking at RFC 2279, we see that the beginning set of ‘1’s with an terminating ‘0’ denotes how many bytes will be used–in this case, 2. Stripping off the initial 110 metadata, we’re left with 00011 from the first byte: that represents the leftmost bits of the actual value.

For the next byte (1010 1001), again from the RFC we see that, for every subsequent byte, 10 will be "prefix" metadata for the actual value. Stripping that off, we’re left with 101001.

Concatenating the actual value bits, we end up with 00011 101001, which is 233 in base-10, or E9 in base-16.

Encoding Identification

The other thing to consider specifically from the UTF-16 value (%FF%FE%E9%00) is from the original RFC, which mentions that there’s no explicit definition of the encoding used, in the encoded value itself. So in this case, iOS is "cheating", giving you an indication of what encoding is used. FF FE is a well-known byte-ordering mark used in UTF-16 encoded files, to denote that UTF-16 is the encoding used. As for E9 00, as mentioned, UTF-16 always uses two bytes. In this case, since all of its data can be represented in 1 byte, the other is simply null.