Question

Using the features currently available in PHP’s intl wrapper for ICU, how would you go about checking for validity of a string’s encoding? (e.g. check for valid UTF-8)

I know it can be done with mbstring, iconv() and PCRE but I’m specifically interested in intl with this question.

Answer 1

I did some digging and found ICU unorm2_normalize() documentation. Its pErrorCode out parameter is interesting. The standard ICU error codes start around line 620 of utypes.h. So I tried this test script:

$s = 'tête-à-tête';
echo "normalizer_normalize(\$s) >> " 
     . var_export(normalizer_normalize($s), 1) . "\n";
$s = "\xFF" . $s;
echo "normalizer_normalize(\$s) >> " 
     . var_export($r=normalizer_normalize($s), 1) . "\n";
if ($r===false)
    echo "normalizer_normalize() error: " 
         . intl_get_error_message() . "\n";
// which outputs:
normalizer_normalize($s) >> 'tête-à-tête'
normalizer_normalize($s) >> false
normalizer_normalize() error: Error converting input string to UTF-16: U_INVALID_CHAR_FOUND

So I guess a test based on that and looking for the following three error codes would be a decent indication of bad UTF-8 encoding:

U_INVALID_CHAR_FOUND Character conversion: Unmappable input sequence.
U_TRUNCATED_CHAR_FOUND Character conversion: Incomplete input sequence.
U_ILLEGAL_CHAR_FOUND Character conversion: Illegal input sequence/combination of input units.

Or when I’m feeling lazy I could just use

normalizer_normalize($s)===false

Btw: I’m confused by this line of the ICU API spec:

pErrorCode Standard ICU error code.
Its input value must pass the
U_SUCCESS() test, or else the function
returns immediately. Check for
U_FAILURE() on output or use with
function chaining. (See User Guide for
details.)

The “the function returns immediately” phrase is encouraging re performance of my test but does “the function” refer to unorm2_normalize() or U_SUCCESS()? Any ideas?