Home/ Questions/Q 7798223
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-01T23:56:27+00:00

Is there a way to recognize if text file is UTF-8 in Python? I

  • 0

Is there a way to recognize if text file is UTF-8 in Python?

I would really like to get if the file is UTF-8 or not. I don’t need to detect other encodings.

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    You mentioned in a comment you only need to detect UTF-8. If you know the alternative consists of only single byte encodings, then there is a solution that often works.

    If you know it’s either UTF-8 or single byte encoding like latin-1, then try opening it first in UTF-8 and then in the other encoding. If the file contains only ASCII characters, it will end up opened in UTF-8 even if it was intended as the other encoding. If it contains any non-ASCII characters, this will almost always correctly detect the right character set between the two.

    try:
    # or codecs.open on Python <= 2.5
    # or io.open on Python > 2.5 and <= 2.7
    filedata = open(filename, encoding='UTF-8').read() 
except:
    filedata = open(filename, encoding='other-single-byte-encoding').read()

    Your best bet is to use the chardet package from PyPI, either directly or through UnicodeDamnit from BeautifulSoup:

    chardet 1.0.1

    Universal encoding detector

    Detects:

    • ASCII, UTF-8, UTF-16 (2 variants), UTF-32 (4 variants)
    • Big5, GB2312, EUC-TW, HZ-GB-2312, ISO-2022-CN (Traditional and Simplified Chinese)
    • EUC-JP, SHIFT_JIS, ISO-2022-JP (Japanese)
    • EUC-KR, ISO-2022-KR (Korean)
    • KOI8-R, MacCyrillic, IBM855, IBM866, ISO-8859-5, windows-1251 (Cyrillic)
    • ISO-8859-2, windows-1250 (Hungarian)
    • ISO-8859-5, windows-1251 (Bulgarian)
    • windows-1252 (English)
    • ISO-8859-7, windows-1253 (Greek)
    • ISO-8859-8, windows-1255 (Visual and Logical Hebrew)
    • TIS-620 (Thai)

    Requires Python 2.1 or later

    However, some files will be valid in multiple encodings, so chardet is not a panacea.

    • 0