Home/ Questions/Q 6235067
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-24T10:36:22+00:00

For example if I know that ć should be ć , how can I

  • 0

For example if I know that ć should be ć, how can I find out the codepage transformation that occurred there?

It would be nice if there was an online site for this, but any tool will do the job. The final goal is to reverse the codepage transformation (with iconv or recode, but tools are not important, I’ll take anything that works including python scripts)

EDIT:

Could you please be a little more verbose? You know for certain that some substring should be exactly. Or know just the language? Or just guessing? And the transformation that was applied, was it correct (i.e. it’s valid in the other charset)? Or was it single transformation from charset X to Y but the text was actually in Z, so it’s now wrong? Or was it a series of such transformations?

Actually, ideally I am looking for a tool that will tell me what happened (or what possibly happened) so I can try to transform it back to proper encoding.

What (I presume) happened in the problem I am trying to fix now is what is described in this answer – utf-8 text file got opened as ascii text file and then exported as csv.

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    It’s extremely hard to do this generally. The main problem is that all the ascii-based encodings (iso-8859-*, dos and windows codepages) use the same range of codepoints, so no particular codepoint or set of codepoints will tell you what codepage the text is in.

    There is one encoding that is easy to tell. If it’s valid UTF-8, than it’s almost certainly no iso-8859-* nor any windows codepage, because while all byte values are valid in them, the chance of valid utf-8 multi-byte sequence appearing in a text in them is almost zero.

    Than it depends on which further encodings may can be involved. Valid sequence in Shift-JIS or Big-5 is also unlikely to be valid in any other encoding while telling apart similar encodings like cp1250 and iso-8859-2 requires spell-checking the words that contain the 3 or so characters that differ and seeing which way you get fewer errors.

    If you can limit the number of transformation that may have happened, it shouldn’t be too hard to put up a python script that will try them out, eliminate the obvious wrongs and uses a spell-checker to pick the most likely. I don’t know about any tool that would do it.

    • 0