Question

The “Incorrect string value error” is raised from MySQLdb.

_mysql_exceptions.OperationalError: (1366, "Incorrect string value: '\\xF0\\xA0\
\x84\\x8E\\xE2\\x8B...' for column 'from_url' at row 1")

But I already set both the connection charset and from url encoding to utf8. It works without problem for millions for records previously.

the value which will cause exception:
I think the issue is related to the special character u’\U0002010e’ (a chinese special character “ㄋ”)

u'http://www.ettoday.net/news/20120227/27879.htm?fb_action_ids=305328666231772&
fb_action_types=og.likes&fb_source=aggregation&fb_aggregation_id=288381481237582
 \u7c89\u53ef\u611b\U0002010e\u22ef http://www.ettoday.net/news/20120221/26254.h
tm?fb_action_ids=305330026231636&fb_action_types=og.likes&fb_source=aggregation&
fb_aggregation_id=288381481237582 \u597d\u840c\u53c8\u22ef'

but this character can be encoded as utf8 in python as well.

>>> u'\U0002010e'.encode('utf8')
'\xf0\xa0\x84\x8e'

So why MySQL cannot accept this character?

Answer 1

The character you are using is outside the BMP, therefore it requires 4 bytes to store. Using the utf8 charset is not enough; you must have MySQL 5.5 or greater and use the utf8mb4 charset instead.