Question

I have a SQLite database with a word list. In a table there is a word list that includes the word “você”. This word has this representation in unicode “voc\U00ea”.

I’ve found out that the same word can have the following representation with the same visual output:

"voc\U00ea",
"voce\U0302"

When I query my db using the second representation it returns blank. Does anyone know a way for the query work using both representations without duplicating the records in the table?

Thanks,
Miguel

Answer 1

These two forms are known as nfc(normal form composed) and nfd(“normal form decomposed”). The letter \U0302 is known as a “combining circumflex”, which modifies a preceding letter.

To cope with this situation, do the following:

1. Pick a normalization. Usually choosing nfc is a good idea. (Although iOS/OS X file system uses nfd.)
2. Before putting a string into the database, always normalize. In iOS, you can use precomposedStringWithCanonicalMapping or precomosedStringWithCompatibilityMapping. To understand the difference between canonical and compatibility mappings, see this description.
3. Before performing a query, always normalize the query to the same normal form.