Question

I’ve been using a couple of methods on a simple class, which have worked ok for me, but I’ve noticed that they are really slow due to strtr() and having a ton of translations defined. Also it is really long and thus, more difficult to maintain and understand.

That said, all of the “bad” examples are solutions to real-world issues which have come up, translating strings into UTF8.

Can anyone tell me of there’s a well known or more efficient means of doing this? (yes, I’ve tried the htmlentities() methods and the iconv()methods but neither really replaced all the funky characters correctly.

Here’s the class I’m using currently:
https://gist.github.com/2559140

Answer 1

mbstring support is now enabled (but not loaded) by default as of PHP 5.4.0. Loading the extension, this lets you do:

<? //PHP 5.4+
$ensureIsUTF8 = static function($data){
    $dataEncoding = \mb_detect_encoding(
        $data,
        ['UTF-8', 'windows-1251', 'iso-8859-1', /*others you encounter*/],
        true
    );

    //UTF-16/32 encoding detection always fails for PHP <= 5.4.1
    //Use detection code copied from PHP docs comments:
    //http://www.php.net/manual/en/function.mb-detect-encoding.php
    if ($dataEncoding === false){

        $UTF32_BIG_ENDIAN_BOM = chr(0x00) . chr(0x00) . chr(0xFE) . chr(0xFF);
        $UTF32_LITTLE_ENDIAN_BOM = chr(0xFF) . chr(0xFE) . chr(0x00) . chr(0x00);
        $UTF16_BIG_ENDIAN_BOM = chr(0xFE) . chr(0xFF);
        $UTF16_LITTLE_ENDIAN_BOM = chr(0xFF) . chr(0xFE);

        $first2 = \substr($data, 0, 2);
        $first4 = \substr($data, 0, 4);

        if ($first4 === $UTF32_BIG_ENDIAN_BOM) {
            $dataEncoding = 'UTF-32BE';
        } elseif ($first4 === $UTF32_LITTLE_ENDIAN_BOM) {
            $dataEncoding = 'UTF-32LE';
        } elseif ($first2 === $UTF16_BIG_ENDIAN_BOM) {
            $dataEncoding = 'UTF-16BE';
        } elseif ($first2 === $UTF16_LITTLE_ENDIAN_BOM) {
            $dataEncoding = 'UTF-16LE';
        } else {
            throw new \Exception('Whoa! No idea what that was.');
        }
    }

    if ($dataEncoding === 'UTF-8'){
        return $data;
    } else {
        return \mb_convert_encoding(
           $data,
           'UTF-8',
           $dataEncoding
        );      
    }
};

$utf8Data = $ensureIsUTF8(\file_get_contents('something'));
$utf8Data = $ensureIsUTF8(\file_get_contents('http://somethingElse'));
$utf8Data = $ensureIsUTF8($userProvidedData);
?>