Question

i have a Perl script which tokenize a string

@s=split /([^a-zA-Z \t\-\'\,\.]+)/, $_[0];  # tokenized with separators

so if i have a string $s="The large [[bear]] is dangerous."

it will return array("The large", " [[", "bear", "]] ", "is dangerous", ".")

But the regex pattern doesn’t seem to work when i used it on a php script.

$tokens = preg_split("/[^a-z \t\-\'\,\.]+/i", $s);

does anyone have any idea about the problem?

Answer 1

Got it to run (Demo):

$s="The large [[bear]] is dangerous.";
$pattern = '/([^a-zA-Z \t\-\\\'\,\.]+)/';    
$tokens = preg_split($pattern, $s, NULL, PREG_SPLIT_DELIM_CAPTURE);    
print_r($tokens);

Output:

Array
(
    [0] => The large
    [1] => [[
    [2] => bear
    [3] => ]]
    [4] =>  is dangerous.
)

Details:

• The pattern must be properly expressed as a string in PHP, mind the string escape sequences. In a single quoted string \' is written as \\\'.
• You want to split including the delimiters, you need to use the PREG_SPLIT_DELIM_CAPTURE flag.

See Single quoted Strings^Docs and preg_split^Docs

Edit: To split at the punctuation, just remove them from the pattern (Demo):

$pattern = '/([^a-zA-Z \t]+)/';