Home/ Questions/Q 951835
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-15T23:47:55+00:00

I’m having issues with Unicode characters in Perl. When I receive data in from

  • 0

I’m having issues with Unicode characters in Perl. When I receive data in from the web, I often get characters like √¢¬Ä¬ú or √¢¬Ç¬¨. The first one is a quotation mark and the second is the Euro symbol.

Now I can easily substitute in the correct values in Perl and print to the screen the corrected words, but when I try to output to a .CSV file all the substitutions I have done are for nothing and I get garbage in my .CSV file. (The quotes work, guessing since it’s such a general character). Also Numéro will give Numéro. The examples are endless.

I wrote a small program to try and figure this issue out, but am not sure what the problem is. I read on another stack overflow thread that you can import the .CSV in Excel and choose UTF8 encoding, this option does not pop up for me though. I’m wondering if I can just encode it into whatever Excel’s native character set is (UTF16BE???), or if there is another solution. I have tried many variations on this short program, and let me say again that its just for testing out Unicode problems, not a part of a legit program. Thanks.

use strict;
use warnings;
require Text::CSV_XS;
use Encode qw/encode decode/;

my $text = 'Numéro Numéro Numéro Orkos Capital SAS (√¢¬Ä¬úOrkos√¢¬Ä¬ù) 325M√¢¬Ç¬¨ in 40 companies headquartered';

print("$text\n\n\n");

$text =~ s/“|”/"/sig;
$text =~ s/’s/'s/sig;
$text =~ s/√¢¬Ç¬¨/€/sig;
$text =~ s/√¢¬Ñ¬¢/®/sig;
$text =~ s/ / /sig;

print("$text\n\n\n");

my $CSV = Text::CSV_XS->new ({ binary => 1, eol => "\n" }) or die "Cannot use CSV: ".Text::CSV->error_diag();

open my $OUTPUT, ">:encoding(utf8)", "unicode.csv" or die "unicode.csv: $!";

my @row = ($text);

$CSV->print($OUTPUT, \@row);
$OUTPUT->autoflush(1);

I’ve also tried these two lines to no avail:

$text = decode("Guess", $text);
$text = encode("UTF-16BE", $text);
Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    So I figured out the answer, the comment from Roland Illig helped me get there (thanks again!). Decoding more than once causes the wide characters error, and therefore should not be done.

    The key here is decoding the UTF-8 Text and then encoding it in MacRoman. To send the .CSV files to my Windows friends I have to save it as .XLSX first so that the coding doesn’t get all screwy again.

    $text =~ s/“|”/"/sig;
$text =~ s/’s/'s/sig;
$text =~ s/√¢¬Ç¬¨/€/sig;
$text =~ s/√¢¬Ñ¬¢/®/sig;
$text =~ s/ / /sig;

$text = decode("UTF-8", $text);

print("$text\n\n\n");

my $CSV = Text::CSV_XS->new ({ binary => 1, eol => "\n" }) or die "Cannot use CSV: ".Text::CSV->error_diag();

open my $OUTPUT, ">:encoding(MacRoman)", "unicode.csv" or die "unicode.csv: $!";
    • 0