Question

I am trying to split a Unicode string into words (simplistic), like this:

print re.findall(r'(?u)\w+', "раз два три")

What I expect to see is:

['раз','два','три']

But what I really get is:

['\xd1', '\xd0', '\xd0', '\xd0', '\xd0\xb2\xd0', '\xd1', '\xd1', '\xd0']

What am I doing wrong?

Edit:

If I use u in front of the string:

print re.findall(r'(?u)\w+', u"раз два три")

I get:

[u'\u0440\u0430\u0437', u'\u0434\u0432\u0430', u'\u0442\u0440\u0438']

Edit 2:

Aaaaand it seems like I should have read docs first:

 print re.findall(r'(?u)\w+', u"раз два три")[0].encode('utf-8')

Will give me:

раз

Just to make sure though, does that sound like a proper way of approaching it?

Answer 1

You’re actually getting the stuff you expect in the unicode case. You only think you are not because of the weird escaping due to the fact that you’re looking at the reprs of the strings, not not printing their unescaped values. (This is just how lists are displayed.)

>>> words = [u'\u0440\u0430\u0437', u'\u0434\u0432\u0430', u'\u0442\u0440\u0438'] 
>>> for w in words:
...     print w # This uses the terminal encoding -- _only_ utilize interactively
... 
раз
два
три
>>> u'раз' == u'\u0440\u0430\u0437'
True

Don’t miss my remark about printing these unicode strings. Normally if you were going to send them to screen, a file, over the wire, etc. you need to manually encode them into the correct encoding. When you use print, Python tries to leverage your terminal’s encoding, but it can only do that if there is a terminal. Because you don’t generally know if there is one, you should only rely on this in the interactive interpreter, and always encode to the right encoding explicitly otherwise.

In this simple splitting-on-whitespace approach, you might not want to use regex at all but simply to use the unicode.split method.

>>> u"раз два три".split()
[u'\u0440\u0430\u0437', u'\u0434\u0432\u0430', u'\u0442\u0440\u0438']

Your top (bytestring) example does not work because re basically assumes all bytestrings are ASCII for its semantics, but yours was not. Using unicode strings allows you to get the right semantics for your alphabet and locale. As much as possible, textual data should always be represented using unicode rather than str.