Question

I want to slice a Unicode string in Ruby.
The slicing should keep the invisible characters intact.

Here’s an example of the input:

Foo\r\n
\r\n
\r\n
Bär 1.234 Foo test\r\n
blub

Which should become:

Array=["Foo\r\n\r\n\r\n","Bär","1.234,"Foo","test\r\n","blub"]

Basically I want to tokenize the string and keep the formatting intact.

When I do something like:

String.split(/ /)

I end up with something like:

Array=["Foo\r\n\r\n\r\nBär","1.234","Foo"]

And, something like:

String.split(/\W/)

kills the formatting.

Answer 1

Instead of using split, use scan for the first part:

text = "Foo\r\n\r\n\r\nBär 1.234 Foo test\r\nblub"
text.scan(/^.+[\r\n]*/)
=> ["Foo\r\n\r\n\r\n", "Bär 1.234 Foo test\r\n", "blub"]

Then conditionally apply your split like this:

text.scan(/^.+[\r\n]*/).map{ |s| s[' '] ? s.split(/ /) : s }.flatten
=> ["Foo\r\n\r\n\r\n", "Bär", "1.234", "Foo", "test\r\n", "blub"]

or:

text.scan(/^.+[\r\n]*/).flat_map{ |s| s[' '] ? s.split(/ /) : s }
=> ["Foo\r\n\r\n\r\n", "Bär", "1.234", "Foo", "test\r\n", "blub"]