Question

I’m opening a text file and removing the first line to prepare it for importing in a database using bulk insert. Here is my code:

string tempFile = Path.GetTempFileName();
using (var sr = new StreamReader("F:\\Upload\\File.txt", System.Text.Encoding.UTF8))
{
    using (var sw = new StreamWriter(tempFile,true, System.Text.Encoding.UTF8))
    {
        string line;
        while ((line = sr.ReadLine()) != null)
        {
            if (line.Substring(0, 8) != "Nr. Crt.")
                sw.WriteLine(line);
        }
    }
}

System.IO.File.Delete("F:\\Upload\\File.txt");
System.IO.File.Move(tempFile, "F:\\Upload\\File.txt");

After this if I open the resulting file, Unicode characters are replaced with other characters. For example strings containing non-breaking space (unicode U+00A0): Value  (note the unicode char ) are transformed in Value� .

How can I avoid this?

Edit:

Notepad++ is set to ‘Encode in UTF-8’
Here is a picture of how it looks :

Answer 1

are transformed in Value�

The byte values for those 3 odd characters are 0xef 0xbd 0xbf. Which is the utf8 encoding for codepoint \ufffd, the replacement character �. Which is used when reading utf encoded text and the text contains an invalid encoding byte sequence.

Pointing squarely at an issue with File.txt, it was probably not encoded in utf-8. If you have no idea what encoding was used for that file then the first guess is to pass Encoding.Default to the StreamReader constructor.