Home/ Questions/Q 8395293
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-09T20:15:47+00:00

See this piece of perl code: #!/usr/bin/perl -w -CS use feature ‘unicode_strings’; open IN,

  • 0

See this piece of perl code:

#!/usr/bin/perl -w -CS

use feature 'unicode_strings';

open IN, "<", "wiki.txt";
open OUT, ">", "wikicorpus.txt";

binmode( IN,  ':utf8' );
binmode( OUT, ':utf8' );

## Condition plain text English sentences or word lists into a form suitable for constructing a vocabulary and language model

while (<IN>) {

  # Remove starting and trailing tags (e.g. <s>)
  # s/\<[a-z\/]+\>//g;

  # Remove ellipses 
  s/\.\.\./ /g;

  # Remove unicode 2500 (hex E2 94 80) used as something like an m-dash between words
  # Unicode 2026 (horizontal ellipsis)
  # Unicode 2013 and 2014 (m- and n-dash)
  s/[\x{2500}\x{2026}\x{2013}\x{2014}]/ /g;

  # Remove dashes surrounded by spaces (e.g. phrase - phrase)
  s/\s-+\s/ /g;

  # Remove dashes between words with no spaces (e.g. word--word)
  s/([A-Za-z0-9])\-\-([A-Za-z0-9])/$1 $2/g;

  # Remove dash at a word end (e.g. three- to five-year)
  s/(\w)-\s/$1 /g;

  # Remove some punctuation
  s/([\"\?,;:%???!()\[\]{}<>_\.])/ /g;

  # Remove quotes
  s/[\p{Initial_Punctuation}\p{Final_Punctuation}]/ /g;

  # Remove trailing space
  s/ $//;

  # Remove double single-quotes 
  s/'' / /g;
  s/ ''/ /g;

  # Replace accented e with normal e for consistency with the CMU pronunciation dictionary
  s/?/e/g;

  # Remove single quotes used as quotation marks (e.g. some 'phrase in quotes')
  s/\s'([\w\s]+[\w])'\s/ $1 /g;

  # Remove double spaces
  s/\s+/ /g;

  # Remove leading space
  s/^\s+//;

  chomp($_);

  print OUT uc($_) . "\n";
#  print uc($_) . " ";
} print OUT "\n";

It seems that there is a non-english character on line 49, namely the line s/?/e/g;.
So when I run this, warning come out that Quantifier follows nothing in regex;.

How can I deal with this problem? How to make perl recognize the character? I have to run this code with perl 5.10.

Another little question is that what is the meaning of the “-CS” in the 1st line.

Thanks to all.

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    I think your problem is that your editor doesn’t handle unicode characters, so the program is trashed before it even gets to perl, and as this apparently isn’t your program, it was probably trashed before it got to you.

    Until the entire tool chain handles unicode correctly, you have to be careful to encode non-ascii characters in a way that preserves them. It’s a pain, and no simple solutions exist. Consult your perl manual for how to embed unicode characters safely.

    • 0