Question

I have a database that stores data as UTF-8, but sometimes some data (outside of my control) is added with unrecognized characters which are stored as question marks (?). Is there any way to configure MySQL to remove those characters instead?

Answer 1

maybe solving the problem instead of working around it will be the better solution. if you just remove the question marks, the stored data wouldn’t be so ugly as it#s now, but it will still be wrong (wrong = not exactly the data the user wanted to save).

this problem sounds like you’ve missed to specify a character encoding somewhere, so that the client’s default settings are used (you havn’t said wich exactly are the clients – maybe the users browser?) wich seems to be utf-8 or something compatible most times – but sometimes there’s a clint with an exotic default charset wich makes you run into problems.

to solve this, simply make sure you’ve set character encoding to utf-8 everywere:

• tell MySQL to use utf-8. to do this, add this to your my.cnf:

  collation_server = utf8_unicode_ci
  character_set_server = utf8
  

• before interacting with mysql, send this two querys:

  SET NAMES 'utf8';
  CHARSET 'utf8';
  

  or, alternatively, let php do this afteropening the connection:

  mysql_set_charset('utf8', $conn);
  

• set UTF-8 as the default charset for your database

  CREATE DATABASE `my_db` DEFAULT CHARACTER SET 'utf8';
  

• do the same for tables:

  CREATE TABLE `my_table` (
    -- ...
  ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
  

• assuming the client is a browser, serve your content as utf-8 and the the correct header:

  header('Content-type: text/html; charset=utf-8');
  

  to be really sure the browser understands, add a meta-tag:

  <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
  

• and, last but not least, tell the browser to submit forms using utf-8

  <form accept-charset="utf-8" ...>