Home/ Questions/Q 181775
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-11T14:49:23+00:00

I have some Perl code that translates new-lines and line-feeds to a normalized form.

  • 0

I have some Perl code that translates new-lines and line-feeds to a normalized form. The input text is Japanese, so that there will be multi-byte characters.

Is it still possible to do this transformation on a byte-by-byte basis (which I think it currently does), or do I have to detect the character set and enable Unicode support? In other words, are the popular encodings (Shift-JIS, EUC-JP, UTF-8, ISO-2022-JP) using bytes as part of their character set that could be mistaken for ASCII control characters?

I need only CR and LF to work.

Update: Added ISO-2022-JP. And that is the one that looks the most troublesome with its funky escape sequences …

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. None of the 4 encodings that you mention (Shift-JIS, UTF-8, EUC-JP, ISO-2022-JP) use the CR or LF character inside Japanese characters. For UTF-8 and EUC-JP, there is no overlap whatsoever between low ascii characters and bytes inside Japanese characters. However, for Shift-JIS, and ISO-2022-JP, there is overlap, but not in the range where you find CR and LF.

    For ISO-2022-JP, First-byte range: 0x21 - 0x7E Second-byte range: 0x21 - 0x7E

    And the escape sequence characters to switch back and forth between various character sets are:

    0x1B, 0x28, 0x24, 0x40, 0x42, and 0x4A

    As you can see, none of the characters used to encode Japanese characters in ISO-2022-JP overlap with CR or LF.

    For Shift-JIS, First-byte range: 0x81 - 0x9F, 0xE0 - 0xEF Second-byte range: 0x40 - 0x7E, 0x80 - 0xFC Half-width katakana: 0xA1 - 0xDF

    Again, there is no overlap with CR and LF.

    • 0