Question

I have a .csv file that I need to upload to the database. There is already an existing PHP script, that parses that data and inserts it into several database tables and this has been working fine. The latest csv file is giving me errors.

I cant get myself to fix this. This is simple English Text with no special characters but I keep getting errors on certain rows.

This is an example of one such row

It is an investment strategy that aims to balance risk and reward

This is the error I get while uploading

ERROR=1366-Incorrect string value: '\xA0strat...' for column 'question' at row 1

The table charset is UTF-8 and the column question is of type TEXT

I’m running the PHP script on Windows command prompt – I have a feeling it has something to do with that. But on all previous occasions, for other CSV files – the php script was running on Windows itself.

Now when I type the above English text in notepad and insert it in CSV file – it works fine – but I have over 500 rows that are giving this error and I cannot keep typing each line in notepad and then pasting it in CSV.

*** EDIT *********
This is the error producing code

foreach($question_array as $question) 
{
   //INSERT QUESTION  table
   mysql_query("INSERT INTO adl_question (question) VALUES ('".$question."')",$con);
   echo 'ERROR='.mysql_errno($con).'-'.mysql_error($con);
   $question_id = mysql_insert_id();
}

Any insights
Thanks

Answer 1

I ran into this problem while doing data conversion recently. What you need to do is use iconv to read it explicitly as WINDOWS-1250 (or whatever character encoding the ‘bad’ text is) and then convert back to UTF-8.

<?php 
$string = iconv('windows-1250', 'utf-8', $badString);
?>

See https://www.php.net/manual/en/function.iconv.php for more information.

If you need to do this for an entire file, you can use this function (that I did not write):

<?php 
function utf8_fopen_read($fileName) {
    $fc = iconv('windows-1250', 'utf-8', file_get_contents($fileName));
    $handle=fopen("php://memory", "rw");
    fwrite($handle, $fc);
    fseek($handle, 0);
    return $handle;
}
?>

And then implement it as so:

<?php 
if (($handle = utf8_fopen_read("mycsv.csv")) !== FALSE) {
    
    while($file = fgetcsv($handle)) {
        $csv[] = $file;
    }
    
    foreach($csv as $row) {
        // run insert here
    }
}
?>