I need to adapt lucene’s StandardTokenizer to some special purposes regarding twitter data. At the moment I use StandardTokenizer to tokenize some tweets, which I want to work at. It worked quite well, but now I want to extend the behaviour (e.g. considere also #hashtags and @somebody, consider smileys :), remove url, …).
Can somebody tell me – or even give me a direction – how I can do this easily? I know, writing my own Tokenizer would be the best choice, but I’m quite new to lucene and I don’t know how to start…
I hope somebody can help me 🙂
Best,
Michael
You can extend the tokenization of StandardTokenizer a great deal by using lucene’s CharFilter apis (and possibly TokenFilters too, depending on how you want the search to work).
Ultimately, if StandardTokenizer is completely different than what you want, then its the wrong tokenizer, but if its close, this can be much easier.
A CharFilter is essentially a FilterReader that lets you modify the text before the Tokenizer runs. It also tracks offset adjustments so that highlighting will still work!
to add CharFilters, the easiest way is to extend ReusableAnalyzerBase and override its initReader method, wrapping the incoming reader with the CharFilters you want.
You might want to look at MappingCharFilter as a start, it lets you define some mappings up front to handle your special twitter syntax. There are some examples/ideas here:
http://markmail.org/message/abo2hysvfy2clxed