Question

I have a text with about 300 – 500 words. Also i got about 200k keywords and i want to know if each of the keywords is contained in the text. A String contains ist quite slow, is there some way to preprocess the String?

I thought about using a SuffixTree but im not sure this is the best choice.

Also, are there any good librarys for this task? semanticdiscoverytoolkit for example has a suffixtree implementation but after adding the string i cant figure out how to look up if a string is contained in the tree.

Greetings,

Nico

Answer 1

you can try the rabin-karp string search algorithm. since you are doing mostly hash (integer) comparisons, the performance is much better than string comparisons.

1. compute the hash of the keyword
2. compute the rolling hash of the text
3. compare these 2 hashes. if they match, perform the actual string comparison.
4. advance the position by 1 character and repeat from step 2 until you reach the end of the text.

as a analogy, the rolling hash is like a “sliding window” that scrolls along the text. the hash comparison is done using the hash of the substring in the “sliding window” against the hash of the keyword.