Home/ Questions/Q 837607
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-15T05:10:05+00:00

How do perl strings represented internally? What encoding is used? How do I handle

  • 0

How do perl strings represented internally? What encoding is used? How do I handle different encodings properly?

I’ve been using perl for quite a long time, but it didn’t include a lot of string handling in different encodings, and when I encountered a minor problem that had something to do with encodings I usually resorted to some shamanic actions.

Until this moment I thought about perl strings as sequences of bytes, which did fit pretty well for my tasks. Now I need to do some processing of UTF-8 encoded file and here starts trouble.

First, I read file into string like this:

open(my $in, '<', $ARGV[0]) or die "cannot open file $ARGV[0] for reading";
binmode($in, ':utf8');

my $contents;

{
    local $/;
    $contents = <$in>;
}

close($in);

then simply print it:

print $contents;

And I get two things: a warning Wide character in print at <scriptname> line <n> and a garbage in console. So I can conclude that perl strings have a concept of “character” that can be “wide” or not, but when printed these “wide” characters are represented in console as multiple bytes, not as single “character”.
(I wonder now why did all my previous experience with binary files worked quite how I expected it to work without any “character” issues).

Why then I see garbage in console? If perl stores strings as character in some known encoding, I don’t think there is a big problem to find out console encoding and print text properly. (I use Windows, BTW).

If perl stores strings as variable-width character sequences (e.g. using same UTF-8 encoding), why is it done this way? From my C experience handling strings is PAIN.

Update.

I use two computers for testing, one runs Windows 7 x64 with English language pack installed, but with Russian regional settings (so I have cp866 as OEM codepage and cp1251 as ANSI) with ActivePerl 5.10.1 x64; another runs Windows XP 32 bit Russian localization with Cygwin Perl 5.10.0.

Thanks to links, now I have much more solid understanding on what’s going on and how things should be done.

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Setting utf8 before reading from the file is good, it automagically decodes the bytes into the internal encoding. (Which is also UTF-8 but you don’t need to know, and shouldn’t rely on.)

    Before printing you need to encode the characters back to bytes.

    use Encode;  
utf8::encode($contents);

    There is also a two argument form of encode, for other encodings than unicode. (That sentence echoes too much, doesn’t it?)

    Here is a good reference. (Would have been more, but it’s my first post.) Check out perlunitut too, and the unicode article on Joel on Software.

    http://www.ahinea.com/en/tech/perl-unicode-struggle.html

    Oh, and it must use multi-byte strings, because otherwise it’s just not unicode.

    • 0