Question

I have had the problem a few times now while working on projects and I would like to know if there’s an elegant solution.

Problem
I am pulling tweets via XML from twitter and uploading them to my DB however when I output them to screen I get these characters:

“moved to dusseldorf.â��”
OR
también

and if I have Russian characters then I get lots of ugly boxes in place.

What I would like is the correct native accents to show under one encoding. I thought was possible with UTF-8.

What I am using

• PHP, MYSQL

After reading in the XML file I am doing the following to cleanse the data:

    $data = trim($data);
    $data = htmlentities($data);
    $data = mysql_real_escape_string($data);

My Database Collation is: utf8_general_ci

Web page character set is: charset=UTF-8

I think it could have something to do with HTML entities but I really appreciate a solution that works across the board on projects.

Thanks in advance.

Answer 1

Replace this line:

$data = htmlentities($data);

With this:

$data = htmlentities($data, null, "UTF-8");

That way, htmlentities() will leave valid UTF-8 characters alone. For more information see the documentation for htmlentities().