Home/ Questions/Q 117769
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-11T03:23:19+00:00

Encoding issues are among the one topic that have bitten me most often during

  • 0

Encoding issues are among the one topic that have bitten me most often during development. Every platform insists on its own encoding, most likely some non-UTF-8 defaults are in the game. (I’m usually working on Linux, defaulting to UTF-8, my colleagues mostly work on german Windows, defaulting to ISO-8859-1 or some similar windows codepage)

I believe, that UTF-8 is a suitable standard for developing an i18nable application. However, in my experience encoding bugs are usually discovered late (even though I’m located in Germany and we have some special characters that along with ISO-8859-1 provide some detectable differences).

I believe that those developers with a completely non-ASCII character set (or those that know a language that uses such a character set) are getting a head start in providing test data. But there must be a way to ease this for the rest of us as well.

What [technique|tool|incentive] are people here using? How do you get your co-developers to care for these issues? How do you test for compliance? Are those tests conducted manually or automatically?

Adding one possible answer upfront:

I’ve recently discovered fliptitle.com (they are providing an easy way to get weird characters written ‘uʍop ǝpısdn’ *) and I’m planning on using them to provide easily verifiable UTF-8 character strings (as most of the characters used there are at some weird binary encoding position) but there surely must be more systematic tests, patterns or techniques for ensuring UTF-8 compatibility/usage.

Note: Even though there’s an accepted answer, I’d like to know of more techniques and patterns if there are some. Please add more answers if you have more ideas. And it has not been easy choosing only one answer for acceptance. I’ve chosen the regexp answer for the least expected angle to tackle the problem although there would be reasons to choose other answers as well. Too bad only one answer can be accepted.

Thank you for your input.

*) that’s ‘upside down’ written ‘upside down’ for those that cannot see those characters due to font problems

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. There is a regular expression to test if a string is valid UTF-8:

    $field =~   m/\A(      [\x09\x0A\x0D\x20-\x7E]            # ASCII    | [\xC2-\xDF][\x80-\xBF]             # non-overlong 2-byte    |  \xE0[\xA0-\xBF][\x80-\xBF]        # excluding overlongs    | [\xE1-\xEC\xEE\xEF][\x80-\xBF]{2}  # straight 3-byte    |  \xED[\x80-\x9F][\x80-\xBF]        # excluding surrogates    |  \xF0[\x90-\xBF][\x80-\xBF]{2}     # planes 1-3    | [\xF1-\xF3][\x80-\xBF]{3}          # planes 4-15    |  \xF4[\x80-\x8F][\x80-\xBF]{2}     # plane 16   )*\z/x;

    But this doesn’t ensure that the text actual is UTF-8.

    An example: The byte sequence for the letter ö (U+00F6) and the corresponding UTF-8 sequence is 0xC3B6.
    So when you get 0xC3B6 as input you can say that it is valid UTF-8. But you cannot surely say that the letter ö has been submitted.
    This is because imagine that not UTF-8 has been used but ISO 8859-1 instead. There the sequence 0xC3B6 represents the character à (0xC3) and ¶ (0xB6) respectivly.
    So the sequence 0xC3B6 can either represent ö using UTF-8 or ö using ISO 8859-1 (although the latter is rather unusual).

    So in the end it’s only guessing.

    • 0