Question

I’m currently working on an application to validate and parse CSV-files.
The CSV files have to be encoded in UTF-8, although sometimes we get files in a false encoding.
The CSV-files most likely contain special characters of the German alphabet (Ä, Ö, Ü, ß) as most of the texts within the CSV file are in German language.

For the part of the validator, i need to make sure, the file is UTF-8 encoded. As long as there are no special characters present, there is most likely no problem with parsing.

What i have tried so far is to read the file as bytes and use some libraries to detect (or guess) the encoding. I tried most of possibilities of this blog post: http://fredeaker.blogspot.com/2007/01/character-encoding-detection.html

But all libraries I tried didn’t return the correct encoding and therefore I couldn’t parse the special characters.

Now to my question:
Is there a way to determine for a given Character Encoding like UTF-8 to detect characters that are not encoded correctly? So basically the characters that are displayed in the (Eclipse) console as quesion marks.

Or is there any other way to correctly determine the character encoding?
I just need to know if it’s UTF-8 or not.

Thank you all in advance for your help! 🙂

Best Regards,
Robert

Answer 1

Byte sequences that cannot be decoded correctly will be replaced with the “replacement character”, \uFFFD, which is displayed like this: �. However, if the output device doesn’t support that character, it is likely to use a question mark (?) instead.

So, after decoding the UTF-8 data into String objects, search for occurrences of \uFFFD.

Alternatively, if you set up an InputStreamReader with an instance of CharsetDecoder that you create yourself, you can get a lot more control. For example, you can specify that if any byte sequences that cannot be decoded, an Exception should be raised. Or you can ignore them. Or you can specify a different character as the replacement character.