Question

I have a column which is primary key for my table, which can have diacritic or normal text.

I have these 2 values:

Håbo and Habo

I want to insert these two column values in my table, but i am getting error:

DBD::mysql::st execute failed: Duplicate entry 'Habo' for key 'PRIMARY'

As i check Håbo is already inserted and it is treating both values same so primary key violation.

My code:

$dbh = DBI->connect($dsn, $user, $pass)
            or die "Unable to connect: $DBI::errstr\n";
$dbh->{'mysql_enable_utf8'}=1;
$dbh->do('SET NAMES utf8');
my $sql = sprintf "INSERT INTO search_term values(%s, %s)", $dbh->quote($search_term), "Data";

My table description

mysql> desc search_term;
+---------------+--------------+------+-----+---------+-------+
| Field         | Type         | Null | Key | Default | Extra |
+---------------+--------------+------+-----+---------+-------+
| search        | varchar(200) | NO   | PRI | NULL    |       |
| site          | varchar(500) | NO   |     | NULL    |       |
+---------------+--------------+------+-----+---------+-------+

How can i make MySQL treat both values as different and insert them?
Any suggestions?

Answer 1

By default, MySQL “helpfully” converts Unicode into their “equivalent” ASCII using something called Unicode Collation. Like a lot of “convenience” features in MySQL, this would be much handier if it told you. I cannot put enough “quotes” around those “words”.

Fortunately the fix is pretty simple, but non obvious. First, change the character set of your tables to UTF8 so text is stored in utf8. Then change the collation to utf8_bin so comparisons will be done exactly. I’m not 100% sure utf8_bin is 100% the right thing, but it works.

ALTER TABLE search_term CONVERT TO CHARACTER SET utf8;
ALTER TABLE search_term COLLATE utf8_bin;

In the future, when you create tables in MySQL, be sure to append CHARACTER SET utf8 to the creation.

CREATE TABLE search_term (
    search varchar(200) primary key,
    site varchar(500)
)
CHARACTER SET utf8
COLLATE       utf8_bin;

Finally, so you don’t have to do this for every table, you can create the database with these defaults already in place.

Here’s a good post on the Unicode gotchas in MySQL and their fixes.

On the Perl side, be sure to use utf8 so the strings you’re passing to MySQL are encoded utf8.

Finally, according to the DBD::mysql manual, you need to turn on UTF8 support when you connect, not after. Would be nice if it issued a warning.

Additionally, turning on this flag tells MySQL that incoming data should be
treated as UTF-8. This will only take effect if used as part of the call to
connect(). If you turn the flag on after connecting, you will need to issue
the command SET NAMES utf8 to get the same effect.

Change your connect to this.

# I threw in RaiseError because its generally a good idea.
my $dbh = DBI->connect($dsn, $user, $pass, { mysql_enable_utf8 => 1, RaiseError => 1 });