Question

I was wondering how PCRE detects word characters from whatever language.
I was testing this string:

"間違つ"

The php file is encoded as UTF-8 and is properly tagged with the Charset=UTF-8 in the Content type tag.

<?php

$string="\xE9\x96\x93\xE9\x81\x95\xE3\x81\xA4"; //Bytestream from "間違つ" 
$string=preg_replace('/\w/','\w',$string);
echo $string;
echo "<br>";


$byte="\xE9"; //I've tried with each byte separately to find word characters
if(preg_match('/\w/',$byte)){
    echo "$byte is a word";
    }
else{ 
    echo "$byte is not a word";
    }
?>

"\xE9" "\xE9" "\xE3" from all the bytes, are words.

It display:

I know why the symbols appear.
The Decoder use the Unicode replacement character, code point FFFD,
as the decoding of an invalid UTF-8 sequence rather than stop processing the text.
There are invalid sequences since one "word character" is replaced by the replacement '\w'
and then it broke the "byte secuence" to show.

So the questions are:

why those characters are matched like words if they aren’t valid UTF-8 sequences?

How to know wich characters are really word characters of all the Unicode Set?

Answer 1

You have to set the u-Flag otherwise it is interpreted as a ISO-8859-1 string.

The following script shows which chars \w match without the u-Flag:

header("Content-Type: text/plain");
$i = 255;
while($i--)
{
    preg_match('/\w/S', chr($i), $m);
    printf("%' 1s \x%s\n", $m[ 0 ], strtoupper(bin2hex($m[ 0 ])));
}

Only [a-zA-Z] are matched by \w if u-Flag is set:

// added 'A' at the beginning and 'B' at the end
preg_match_all('/\w/u', "A\xE9\x96\x93\xE9\x81\x95\xE3\x81\xA4B", $m);
print_r($m);

Attention: If the u-Flag is present preg_* will silently fail to parse the string if it contains non-unicode-chars (e.g. \x80-\xFF).