Question

I am working on a data mining algorithm where I need to tokenize the string using multiple words. I have a separate file which contain all the stopwords. What I need to do is to tokenize the input string by any of the word (stopword) working as delimiter.
For eg.
If the file contains stopwords as
a
is
and
of
that

and the input string comes to be
“a computer cluster consists of a set of loosely connected computers that work together”
the output comes to be
computer cluster consists
set
loosely connected computers
work together

Checking string against all the stopwords recursively would be very time consuming? Is there any good method for this?

Answer 1

Construct a regular expression of the form

delim1|delim2|delim3

then use String‘s split() method to split the text by any of the delimiters.

In order to construct the regexp, read each delimiter, and pass it to Pattern.quote before appending to the regex that you build. This would let your delimiters use regex metacharacters as well.