Question

I encountered a particular string (it’s not completely printable, but you can see it below) that causes a htmlspecialchars() to return a zero-length string. Is there any way this can be fixed?

$Stmnt = 'SELECT subject_name FROM bans WHERE id = 2321';
$Fetch = $Conn->query($Stmnt);
if(!$Fetch)
    die('Could not query DB');
while($Row = $Fetch->fetch_array(MYSQLI_ASSOC))
{
    $RawName = $Row['subject_name'];
    $RawLen = strlen($RawName);
    echo('RAW NAME: ['.$RawName.']'.', LENGTH: ['.$RawLen.']'.'<br />');
    for($i = 0; $i < $RawLen; $i++)
        echo('CHAR '.$i.' = ['.$RawName[$i].'] (ORD: '.ord($RawName[$i]).')<br />');

    $CleanName = htmlspecialchars($RawName, ENT_QUOTES, 'UTF-8');
    $CleanLen = strlen($CleanName);
    echo('CLEAN NAME: ['.$CleanName.']'.', LENGTH: ['.$CleanLen.']'.'<br />');
    for($i = 0; $i < $CleanLen; $i++)
        echo('CHAR '.$i.' = ['.$CleanName[$i].'] (ORD: '.ord($CleanName[$i]).')<br />');
}
$Fetch->close();
echo('DONE');

Output:

RAW NAME: [━═★ Coммander Fι5н �], LENGTH: [31]
CHAR 0 = [�] (ORD: 226)
CHAR 1 = [�] (ORD: 148)
CHAR 2 = [�] (ORD: 129)
CHAR 3 = [�] (ORD: 226)
CHAR 4 = [�] (ORD: 149)
CHAR 5 = [�] (ORD: 144)
CHAR 6 = [�] (ORD: 226)
CHAR 7 = [�] (ORD: 152)
CHAR 8 = [�] (ORD: 133)
CHAR 9 = [ ] (ORD: 32)
CHAR 10 = [C] (ORD: 67)
CHAR 11 = [o] (ORD: 111)
CHAR 12 = [�] (ORD: 208)
CHAR 13 = [�] (ORD: 188)
CHAR 14 = [�] (ORD: 208)
CHAR 15 = [�] (ORD: 188)
CHAR 16 = [a] (ORD: 97)
CHAR 17 = [n] (ORD: 110)
CHAR 18 = [d] (ORD: 100)
CHAR 19 = [e] (ORD: 101)
CHAR 20 = [r] (ORD: 114)
CHAR 21 = [ ] (ORD: 32)
CHAR 22 = [F] (ORD: 70)
CHAR 23 = [�] (ORD: 206)
CHAR 24 = [�] (ORD: 185)
CHAR 25 = [5] (ORD: 53)
CHAR 26 = [�] (ORD: 208)
CHAR 27 = [�] (ORD: 189)
CHAR 28 = [ ] (ORD: 32)
CHAR 29 = [�] (ORD: 226)
CHAR 30 = [�] (ORD: 148)
CLEAN NAME: [], LENGTH: [0]
DONE

Answer 1

I understand now why it’s returning a zero-length string. Sorry for asking this question. I should have researched more before posting. Anyway, the answer is the following:

On the PHP manual page for htmlspecialchars:

If the input string contains an invalid code unit sequence within the given encoding an empty string will be returned, unless either the ENT_IGNORE or ENT_SUBSTITUTE flags are set.

Then I ask myself what is “invalid” about this string? On the Wiki page for UTF-8 it gives a good diagram of UTF-8 encoding. All codepoints representing “plain text ASCII” would be 0-127 (the MSB in the byte is always 0).

If a byte’s MSB is 1 (decimal 128 to 255) it tells a UTF-8 compliant parser that the codepoint consists of a multi-byte chain. And the next byte’s first two Most-Significant-Bits must be a 1 followed by a 0.

Obviously in this string, there is a case where one byte is over 127 and the following byte does not begin with a 1 & 0. Therefore it is invalid UTF-8 encoding.

Thanks for this SO post for the resolution, which in my opinion, is to use the ENT_SUBSTITUTE flag (or I suppose ENT_IGNORE if you are sure that deleting these non-conforming bytes won’t be a security issue).