Question

Source

RegexOptions.IgnoreCase is more expensive than I would have thought (eg, should be barely measurable)

Assuming that this applies to PHP, Python, Perl, Ruby etc as well as C# (which is what I assume Jeff was using), how much of a slowdown is it and will I incur a similar penalty with /[a-zA-z]/ as I will with /[a-z]/i ?

Answer 1

Yes, [A-Za-z] will be much faster than setting the RegexOptions.IgnoreCase, largely because of Unicode strings. But it’s also much more limiting — [A-Za-z] does not match accented international characters, it’s literally the A-Za-z ASCII set and nothing more.

I don’t know if you saw Tim Bray’s answer to my message, but it’s a good one:

One of the trickiest issues in internationalized search is upper and lower case. This notion of case is limited to languages written in the Latin, Greek, and Cyrillic character sets. English-speakers naturally expect search to be case-insensitive if only because they’re lazy: if Nadia Jones wants to look herself up on Google she’ll probably just type in nadia jones and expect the system to take care of it.

So it’s fairly common for search systems to “normalize” words by converting them all to lower- or upper-case, both for indexing and queries.

The trouble is that the mapping between cases is not always as straightforward as it is in English. For example, the German lower-case character “ß” becomes “SS” when upper-cased, and good old capital “I” when down-cased in Turkish becomes the dotless “ı” (yes, they have “i”, its upper-case version is “İ”). I have read (but not verified first-hand) that the rules for upcasing accented characters such “é” are different in France and Québec. One of the results of all this is that software such as java.String.toLowerCase() tends to run astonishingly slow as it tries to work around all these corner-cases.

http://www.tbray.org/ongoing/When/200x/2003/10/11/SearchI18n