Question

I’m new to Ruby, Regex and Stackoverflow. xD Here’s my problem:

I want to use regex to extract phrases consisting of consecutive words with standard ASCII characters apart from the others in Vietnamese texts.

In another word, phrases with \w characters only, for example:

Mình rất thích con Sharp này (mặc dù chưa xài bao h nhưng chỉ nghe các
pác nói mình đã thấy phê lòi mắt rồi). Các bạn cho mình hỏi 1 câu (các
bạn đừng chê mình ngu nhé tội nghiệp mình) : cái máy này đem sang Anh
dùng mạng Vodafone là dùng vô tư ah`? Nếu dùng được bên Anh mà không
phải chọc ngoáy j thì mình mua một cái

Don’t care about its meaning, what I want to achieve is an array of hashes containing the results with 2 pairs: value => the value of extracted phrases, starting_position => the position of the first character.

According to the example about, it should be like this:
[{:value=>”con Sharp”, :starting_position => 16}, {:value=>”bao h”, :starting_position => blah blah}…]

This means that all words containing \W characters, such as “mình”, “rất”, “thích”, etc. are rejected.

Trying above example with this regex on rubular.com for Ruby 1.9.2:

\b[\w|\s]+\b

I nearly got my desired phrases (except space-only ones), but it seems not working on my Ruby, which is also 1.9.2p290, using Win 7 64-bit.

Any ideas would be highly appreciated. Thank you beforehand.

Answer 1

According to rubular, it looks like \w matches all ascii letters and numbers (and underscored), but \b is working well for all Unicode letters. That is a little confusing.

What you want, however, are all sequences of ASCII words. This should match them:

/\b[a-z]+\b(?:\s+[a-z]+)*\b/i

Working example: http://www.rubular.com/r/1iewl7MpJe

A quick explanation:

• \b[a-z]+\b – first ASCII word.
• (?:\s+[a-z]+) – any number of spaces and words – at least one space and one letter each time.
• \b – to assure the last word doesn’t end in the middle of another word, like n in "con Sharp này".

I’m not sure about getting an hash, but you can get all MatchDatas, similar to:
How do I get the match data for all occurrences of a Ruby regular expression in a string?

s = "hello !@# world how a9e you"
r = /\b[a-z]+\b(?:\s+[a-z]+)*\b/i

matches = s.to_enum(:scan, r).map { Regexp.last_match }
           .map {|match| [match.to_s(), match.begin(0)]}
puts matches 

Here’s an example on ideone: http://ideone.com/YRZE5