Home/ Questions/Q 6058271
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-23T08:34:52+00:00

I am building a data import tool for the admin section of a website

  • 0

I am building a data import tool for the admin section of a website I am working on. The data is in both French and English, and contains many accented characters. Whenever I attempt to upload a file, parse the data, and store it in my MySQL database, the accents are replaced with ‘?’.

I have text files containing data (charset is iso-8859-1) which I upload to my server using CodeIgniter’s file upload library. I then read the file in PHP.

My code is similar to this:

$this->upload->do_upload()
$data = array('upload_data' => $this->upload->data());

$fileHandle = fopen($data['upload_data']['full_path'], "r");

while (($line = fgets($fileHandle)) !== false) {
    echo $line;
}

This produces lines with accents replaced with ‘?’. Everything else is correct.

If I download my uploaded file from my server over FTP, the charset is still iso-8850-1, but a diff reveals that the file has changed. However, if I open the file in TextEdit, it displays properly.

I attempted to use PHP’s stream_encoding method to explicitly set my file stream to iso-8859-1, but my build of PHP does not have the method.

After running out of ideas, I tried wrapping my strings in both utf8_encode and utf8_decode. Neither worked.

If anyone has any suggestions about things I could try, I would be extremely grateful.

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    It’s Important to see if the corruption is happening before or after the query is being issued to mySQL. There are too many possible things happening here to be able to pinpoint it. Are you able to output your MySql to check this?

    Assuming that your query IS properly formed (no corruption at the stage the query is being outputted) there are a couple of things that you should check.

    1. What is the character encoding of the database itself? (collation)

    2. What is the Charset of the connection – this may not be set up correctly in your mysql config and can be manually set using the ‘SET NAMES’ command

    In my own application I issue a ‘SET NAMES utf8’ as my first query after establishing a connection as I am unable to change the MySQL config.

    See this.
    http://dev.mysql.com/doc/refman/5.0/en/charset-connection.html

    Edit: If the issue is not related to mysql I’d check the following

    1. You say the encoding of the file is ‘charset is iso-8859-1’ – can I ask how you are sure of this?

    2. What happens if you save the file itself as utf8 (Without BOM) and try to reprocess it?

    3. What is the encoding of the php file that is performing the conversion? (What are you using to write your php – it may be ‘managing’ this for you in an undesired way)

    4. (an aside) Are the files you are processing suitable for processing using fgetcsv instead?
      http://php.net/manual/en/function.fgetcsv.php

    • 0