Question

I am writing a piece of code which will extract words from running text. This text can contain delimiters like \r,\n etc. which might be there in text.

I want to discard all these delimiters and only extract full words. How can I do this with Python? any library available for crunching text in python?

Answer 1

Assuming your definition of “word” agrees with that of the regular expression module (re), that is, letters, digits and underscores, it’s easy:

import re
fullwords = re.findall(r'\w+', thetext)

where thetext is the string in question (e.g., coming from an f.read() of a file object f open for reading, if that’s where you get your text from).

If you define words differently (e.g. you want to include apostrophes so for example “it’s” will be considered “one word”), it isn’t much harder — just use as the first argument of findall the appropriate pattern, e.g. r"[\w']+" for the apostrophe case.

If you need to be very, very sophisticated (e.g., deal with languages that use no breaks between words), then the problem suddenly becomes much harder and you’ll need some third-party package like nltk.