Question

I have a look up table for isAlpha.

for (int i = 0; i <= UCHAR_MAX; ++i)
p.isalphaLUT[i] = isalpha(i);

where isalphaLUT is a char array…The problem is isalphaLUT[i] where i is a character outside the ASCII range, (it prints 4294967168 when i attempt to get the equivalent ASCII value). I tried setting all ASCII ranges above 127 equal to 0, but that does not work. The character in question is this ö.

Answer 1

The correct way to test if a character is a letter is to test that it is in one of the letter categories: Lu, Ll, Lt, Lm, or Lo. You can use the ICU library from IBM to do this, it is a quite popular library for handling Unicode.

http://icu-project.org/apiref/icu4c/uchar_8h.html

You can also use the u_isalpha function from ICU directly, or u_charType to determine the category of a character. Note that the term “letter” is preferred to “alpha”, since there are many non-alphabetic “letters” in Unicode (such as Chinese characters).

However, you must decode the character first. If you are using an array of char, then your encoding could be ASCII, LATIN-1, Windows 1252, UTF-8, or any of a zillion other encodings. If you access a char directly, it could be signed or unsigned depending on your platform, which is why you’d get an obviously wrong number like 4294967168 — which is just what happens when the byte 0x80 is interpreted as a signed char and then cast to an unsigned int.

Using a lookup table is a very poor choice for this kind of task because the table would have to be so large — about 700k. Instead, I recommend either using ICU or creating a table of ranges of characters and performing a binary search in the table. This can be quite efficient.

I am working on a tool to create exactly these kinds of tables. The tool is currently not ready for production, but if you’re adventurous you can use it, and the README has examples of how to use it.

https://github.com/depp/uniset