Question

I have a little problem, I am building a database from CSV files using a Java application connected to the mySQL database.

CSV is ISO-8859-1 encoded.
It is read via a buffered file reader and parsed with String methods.
Then the strings are introduced to mySQL via JDBC driver.

Problem is: accents (this is a french application) are lost in the transfer. In the mySQL database, they are in an unidentified format which is not UTF-8 neither Latin-1…

My hypothesis is that the Strings are encoded weirdly and keep this encoding when reinserted. How can-I enforce the charset for an INSERT statement in Java?

Answer 1

You need to ensure that you read the CSV using InputStreamReader with the proper charset (which is the one of the file itself, which is in this particular case thus ISO-8859-1).

BufferedReader reader = new BufferedReader(new InputStreamReader(input, "ISO-8859-1"));

You also need to ensure that the JDBC connection string contains a characterEncoding parameter with the proper charset (which is the one the table is been created with, which you have yet to figure out on the MySQL database). If it appears to be an Unicode charset, then you need to add the parameter useUnicode=true as well.

String url = "jdbc:mysql://localhost:3306/dbname?characterEncoding=UTF-8&useUnicode=true";

Your next question shall probably be How do I determine which charset my DB table is using?. You can do this using the SHOW command. It’ll contain information about the charset.

SHOW CREATE DATABASE dbname; -- shows CREATE DATABASE statement.
SHOW CREATE TABLE dbname.tblname; -- shows CREATE TABLE statement.

That said, unrelated to the problem, are you aware that MySQL offers builtin CSV import facilities and that you thus don’t necessarily need Java/JDBC for this? Checkout the LOAD DATA INFILE command. You can specify the CSV file’s charset as command argument and MySQL will worry about the correct conversion itself.