Home/ Questions/Q 7521927
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-30T02:22:17+00:00

I just learned about character sets today, so forgive the newb factor if this

  • 0

I just learned about character sets today, so forgive the newb factor if this is confusing. Please ask for clarification if it’s needed.

I wrote a program in php which recursively goes through the files in a folder and stores the file names in a database. The file names are then all exported from the database in json format using the json_encode($array) function.

However this function only works with UTF-8 encoded data. And since a few of the key-value pairs in the json export have the value of null, I’m lead to believe that those strings of filenames taken from the database are in fact not utf-8.

I’ve ensured that all the data going in and out of the the database is utf-8 by setting the defaults to utf-8 in my.cnf and restarting mysql from the command line using service mysql restart

[client]
default-character-set=utf8

[mysqld]
default-character-set = utf8

I then created my database, the table and all the columns in the table and confirmed that the database, table and all the columns are in fact utf-8

Checks if database is utf-8

SELECT default_character_set_name FROM information_schema.SCHEMATA S
WHERE schema_name = "schemaname";

Checks if table is utf-8 

SELECT CCSA.character_set_name FROM information_schema.`TABLES` T,
       information_schema.`COLLATION_CHARACTER_SET_APPLICABILITY` CCSA
WHERE CCSA.collation_name = T.table_collation
  AND T.table_schema = "schemaname"
  AND T.table_name = "tablename";

Checks if field is utf-8 

SELECT character_set_name FROM information_schema.`COLUMNS` C
WHERE table_schema = "schemaname"
  AND table_name = "tablename"
  AND column_name = "columnname";

There’s this file that has the characters –µ–ª–∫—É–Ω—á–∏–∫ in the file name. When it’s stored in the database the values appear as –©–µ–ª–â’.

Per my database settings, are all the strings going in and out of my database utf-8?

What can I do to ensure the data I am SELECT’ing from the database is utf-8, so I can perform json_encode($array)? (NOTE: this function only works on utf-8 encoded data)

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Unfortunately I don’t know how you can ensure everything coming out is UTF-8 (now I’m curious too!), but a starting point would be trying this in your PHP:

    $encodedNames = array();
$errors = array();

// Loop through all of the filenames
foreach($filenames as $filename)
{
  // Check if it's UTF-8 encoded
  if('UTF-8' === mb_detect_encoding($filename, 'UTF-8', true))
  {
    $encodedNames[] = $filename;
  }
  else
  {
    $errors[] = $filename;
  }
}

// json_encode the UTF-8 filenames
$jsonString = json_encode($encodedNames);

// Log the other filenames here so you can deal with them later...

    http://php.net/manual/en/function.mb-detect-encoding.php

    • 0