Question

I am reading a text file which I guess is encoded in utf-8. Some lines can only be decoded as latin-1 though. I would say this is very bad practice, but nevertheless I have to cope with it.

I have the following questions:

First: how can I “guess” the encoding of a file? I have tried enca, but in my machine:

enca --list languages
belarussian: CP1251 IBM866 ISO-8859-5 KOI8-UNI maccyr IBM855 KOI8-U
  bulgarian: CP1251 ISO-8859-5 IBM855 maccyr ECMA-113
      czech: ISO-8859-2 CP1250 IBM852 KEYBCS2 macce KOI-8_CS_2 CORK
   estonian: ISO-8859-4 CP1257 IBM775 ISO-8859-13 macce baltic
   croatian: CP1250 ISO-8859-2 IBM852 macce CORK
  hungarian: ISO-8859-2 CP1250 IBM852 macce CORK
 lithuanian: CP1257 ISO-8859-4 IBM775 ISO-8859-13 macce baltic
    latvian: CP1257 ISO-8859-4 IBM775 ISO-8859-13 macce baltic
     polish: ISO-8859-2 CP1250 IBM852 macce ISO-8859-13 ISO-8859-16 baltic CORK
    russian: KOI8-R CP1251 ISO-8859-5 IBM866 maccyr
     slovak: CP1250 ISO-8859-2 IBM852 KEYBCS2 macce KOI-8_CS_2 CORK
    slovene: ISO-8859-2 CP1250 IBM852 macce CORK
  ukrainian: CP1251 IBM855 ISO-8859-5 CP1125 KOI8-U maccyr
    chinese: GBK BIG5 HZ
       none:

Which is not enough to detect latin-1. By the way: how can I increase the list of available languages for enca?

Second: is it possible to have a file partially encoded in different encodings? I think that, from a technical point of view, this is very much possible: just use different codecs for different parts of the file. Have you ever seen this? How would you handle reading such a file? Would you try to decode each line with a different encoding? How would you go about this? I am actually using python, and an example of handling multiple-encoding would be appreciated.

Third: how do you usually go about files for which you do not know the encoding that they are using? I encounter this problem often when downloading text files from different sources. They are usually CSV files exported from spreadsheets (I guess excel), but depending on the exporting parameters, they are using one encoding or another, without hints about which one has been chosen, so I am forced to guess.

Answer 1

1. Guess the text encoding a file – in what? Python, Perl, Java, C? Note that this sometimes produces unexpected results (e.g. the “bush hid the facts” bug)
2. Sometimes a file will have text encoded in different encodings. Either the file format specifies different encodings for different parts in the file, or the format permits specifying an encoding switch. For example in MIME the headers keys are ascii, the header values can have embedded sections encoded in other encodings, and the encoding of the content is declared in the Content-Type header.
3. If you don’t know the encoding you just have to read bytes from the file (in a binary safe manner) to a buffer until you can determine the encoding.