Question

I am trying to use a preg_replace to eliminate the Japanese full-width white space “　” from a string input but I end up with a corrupted multi-byte string.

I would prefer to preg_replace instead of str_replace.
Here is a sample code:

$keywords = '　ラメ単色';
$keywords = str_replace(array(' ', '　'), ' ', urldecode($keywords)); // outputs :'ラメ単色'

$keywords = preg_replace("@[ 　]@", ' ',urldecode($keywords)); // outputs :'�� ��単色'

Anyone has any idea as to why this is so and how to remedy this situation?

Answer 1

Add the u flag to your regex. This makes the RegEx engine treat the input string as UTF-8.

$keywords = preg_replace("@[ 　]@u", ' ',urldecode($keywords));
// outputs :'ラメ単色'

CodePad.

The reason it mangles the string is because to the RegEx engine, your replacement characters, 20 (space) or e3 80 80 (IDEOGRAPHIC SPACE) are not treated as two characters, but separate bytes 20, e3 and 80.

When you look at the byte sequence of your string to scan, we get e3 80 80 e3 83 a9 e3 83 a1 e5 8d 98 e8 89 b2. We know the first character is a IDEOGRAPHIC SPACE, but because PHP is treating it as a sequence of bytes, it does a replacement individually of the first four bytes, because they match individual bytes that the regex engine is scanning.

As for the mangling which results in the � (REPLACEMENT CHARACTER), we can see this happens because the byte e3 is present further along in the string. The e3 byte is the start byte of a three byte long Japanese character, such as e3 83 a9 (KATAKANA LETTER RA). When that leading e3 is replaced with a 20 (space), it no longer becomes a valid UTF-8 sequence.

When you enable the u flag, the RegEx engine treats the string as UTF-8, and won’t treat your characters in your character class on a per-byte basis.