Home/ Questions/Q 8617751
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-12T05:56:18+00:00

When tokenizing a string of text, I need to extract the indexes of the

  • 0

When tokenizing a string of text, I need to extract the indexes of the tokenized words. For example, given:

"Mary didn't kiss John"

I would need something like:

[(Mary, 0), (did, 5), (n't, 8), (kiss, 12), (John, 17)]

Where 0, 5, 8, 12 and 17 correspond to the index (in the original string) where the token began. I cannot rely on just whitespace, since some words become 2 tokens. Further, I cannot just search for the token in the string, since the word likely will appear multiple times.

One giant obstacle is that I’m working with “dirty” text. Here is a real example from the corpus, and its tokenization:

String:

The child some how builds a boaty  c capable of getting scrtoacross the sea, even after findingovercoming many treachrous rous obsittalcles.

Tokens:

The, child, some, how, builds, a, boaty, , , c, , capable, of, getting, scrto, , across, the, sea, ,, even, after, finding, , , , , overcoming, many, treachrous, rous, obsittalcles, .

I’m currently using OpenNLP to tokenize the text, but am ambivalent about which API to utilize for tokenization. It does need to be Java, though, so (unfortunately) Python’s NLTK is out of the picture.

Any ideas would be greatly appreciated! Thanks!

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    OpenNLP will return the offsets using the method Tokenizer.tokenizePos(String s), see the OpenNLP API for TokenizerME as an example for one the implemented tokenizers. Each Span returned contains the start and end positions of the token.

    Whether you decide to use UIMA is really a separate question, but OpenNLP does provide UIMA annotators for their tokenizers that use tokenizePos(). However, if you just want to tokenize a string, UIMA is definitely overkill…

    • 0