Question

So I have programmed a crawler to scrape information and data from a website with charset utf8. But when I tried to store the contents into MySQL, some special characters, such as Spanish letters), did not show correctly in MySQL.

Here is what I have done:

1. Put header("Content-Type: text/html; charset=utf-8") in PHP
2. Set all charset in MySQL into utf8-unicode-ci
3. Have $conn->query("SET NAMES 'utf8'") this upon connection
4. Double checked that the html I parsed was encoded in utf-8

So what are some potentially problems here?

Answer 1

Maybe you coded your crawler using functions which are not supposed to manage multi-byte characters.
For example strlen instead of mb_strlen.

Try putting:

mb_internal_encoding("UTF-8");

as first line of your php coce, and then check if you have to convert some functions in their respective mb version.
Have a look at multibyte string reference

As a last chance you may play with iconv function just before inserting the string into mysql.
Something as:

$utf8_string = iconv(iconv_get_encoding($string), "UTF-8", $string);

should do the trick