Question

Here is an excellent question and the wonderful tchrist‘s answer with 7+24+52 advices&comments how to make an perl program utf8 safe.

But here is 19k CPAN modules. What is possible to do for differentiating “good” and “bad” ones? (from the utf8’s point of view)

For example: File::Slurp if you will read the file with

#use strict encoding warnings utf8 autodie... etc....
my $str = read_file($file, binmode => ':utf8');

you will get different results based on command line switches, and perl -CSDA will not work. Sad. (Yes, i know than Encode::decode(“utf8”, read_file($file, binmode => ‘:raw’)); will help, but SAD anyway.

My questions:

• is here any preferred way, how to test/classify what CPAN modules are utf8 safe/ready/correct?
• is here some Test::something already done for utf8 testing?
• is here something like Perl::Critic for utf8 – what will check the module source for possible utf8 incorrectness? (because manually checking sources for 7+24+52 things i cannot classify as the “easy way to programming”)
• or any other way? 🙂

I understand, than much of CPAN modules simply does not need to know about utf8. But here are zilion others what should.

Please, don’t misunderstand me. I love Perl language. I know than perl has extremely powerful utf8 capability. (especially 5.14). The above was not mean as perl critique – but me (and probably some others too) need to know what CPAN modules are OK, and how classify them…)

When doing development using several CPAN modules, and initially everything goes well but in the final testing, you find that some modules does not support utf8, and therefore part of your work is useless – that really can cause a bit disillusionment. 🙁

Edit:

I understand than all complicated things around the unicode has two roots:

1. unicode itself – as tchrist excellently analyzed some of problematic points
2. perl – simple can’t break all working modules, live servers etc – so need maintaining backward compatibility.

My only hope: perl6. Is is an totally new and different language. Don’t need maintaining any backward compatibility. So I hope, in perl6 will be default some things what is not possible do in perl5 and all utf8 things will be much more intuitive.

But, back to modules: @daxim told: “Authors won’t even reveal whether their module is taint-safe, and this feature exists for decades!” – and this is a catastrophe. Maybe (big maybe, and honestly haven’t idea how to do it), but maybe we arrived to the time, when need put much-much harder restrictions into CPAN submissions.

At one side i’m really very happy with volunteer works of CPAN authors. At the other side, publishing source code is not only like a free speech “right” – but should obey some rules too.

I understand, than is is nearly impossible make any “revolution”, but we probably need some “evolution”. Maybe flag any CPAN module what is not utf8 safe. Flag all what are not taint safe. Flag (like here in SO) what module does not meet the minimal coding standards and remove them. Maybe I’m an idealist and/or naive. 🙂

Answer 1

Chill, the situation is less dire than you’re thinking. No one except tchrist operates on this level of Unicode correctness, also see Aristotle’s recent commentary. As with all things, you get 80% of the way with 20% of the effort. This base effort, namely getting the topic of character encoding right, is well documented; and jrockway repeats it in his answer in that thread.

Replies to your specific questions:

1. No, there isn’t. There is no concerted effort to collect this information in a central place. The Perl 5 wiki could be used to document problematic modules, Juerd already discusses some in uniadvice. I would really like to see a statement from each module author in their documentation that “this module DTRT w.r.t. encoding”, but I don’t see it happening. Authors won’t even reveal whether their module is taint-safe, and this feature exists for decades!

2. encoding::warnings can be used to smoke out unintended upgrades. I mention it in the work-flow of Checklist for going the Unicode way with Perl

3. You can’t do that with Perl::Critic or static analysis. I see no other way than knowledgable people poking at the module with pointy characters until it falls apart (or not), like mirod just commented.