Question

I’m thinking about a special regexp problem in PHP, but I can’t find an solution.
I try to split some text into terms to get simple words, numbers and web addresses.

So i decided to split on every non alphanumeric character ( \w ).

To work with different languages, I use \w with additional letters, like Ää éèÈ and so on.

Example:

20,000 15.20 This is at Text. Right?!
www.google.com Jean Béraud

Until now, i use the following regexp to split the text:

[^\w(äÄüÜöÖßèé)]

http://regexr.com?2uq98

Which works well in 80% of cases, but splits 20,000 into 20 and 000 also http://www.google.com into www google com

So i tried to hold the numbers together, but still split on points, like Text. to get Text

To match 15.20, the following works: (\d+\.\d+), but how do I combine the negation with the other regexp string? The following will not work: (\d+\.\d+)|[^\w(äÄüÜöÖßèé)]?
And: how do I handle the web address?

Answer 1

Something like this?

$result = preg_split('/\W*\s+/', $string, NULL, PREG_SPLIT_NO_EMPTY);

Demo, Result:

array(10) {
  [0]=>
  string(6) "20,000"
  [1]=>
  string(5) "15.20"
  [2]=>
  string(4) "This"
  [3]=>
  string(2) "is"
  [4]=>
  string(2) "at"
  [5]=>
  string(4) "Text"
  [6]=>
  string(5) "Right"
  [7]=>
  string(14) "www.google.com"
  [8]=>
  string(4) "Jean"
  [9]=>
  string(7) "Béraud"
}

Q: Why does \w matches é in my example?

A: That’s based on the local of the system the PCRE library is used on, from the PHP Manual:

A “word” character is any letter or digit or the underscore character, that is, any character which can be part of a Perl “word”. The definition of letters and digits is controlled by PCRE’s character tables, and may vary if locale-specific matching is taking place. For example, in the “fr” (French) locale, some character codes greater than 128 are used for accented letters, and these are matched by \w.

Alternatively it might be helpful to specify the regex as working with UTF-8:

$result = preg_split('/\W*\s+/u', $string, NULL, PREG_SPLIT_NO_EMPTY);

Ensure $string is UTF-8 encoded. As UTF-8 is international, specific locale settings might not needed to be taken into account. Give it a try.