Question

I would like to exclude full stop . from string tokens like in the following case:

 I go to school.

Would like to generate tokens as:

        [ I] [go] [to] [school]

I used stringTokenizer as:

List<String> listStr = new ArrayList<String>();
StringTokenizer strToken = new StringTokenizer(str);                    
int i =0;
while(strToken.hasMoreTokens()){
    listStr.add(i, strToken.nextToken());
    i=i+1;
}   

But the last token is school. which i don;t want. How would i do it?

Answer 1

The StringTokenizer has a constructor that takes another argument: The characters that act as delimiters. The following code illustrates this:

StringTokenizer tokenizer = new StringTokenizer("I go to school.", " \t\r\n.");
while(tokenizer.hasMoreElements()) {
    System.out.println(tokenizer.nextElement());
}

This is the output, which should be what you want:

I
go
to
school