Question

I’m creating a dictionary app that searches both a word and its definition (two separate Fields). However, using a StandardAnalyzer, no search results are returned when the search string contains whitespace or special characters.

For example, in my dummy dictionary data, searching “lorem” searches all words that have “lorem” in their definitions, but searching for “lorem ipsum” returns nothing, even though most of my dummy words have a lorem ipsum in the definition.

Also, searching for words like “make-believe” only returns results when typing “make,” but as soon as I include the dash, nothing is returned.

I want to include characters like whitespace, dashes, commas, whatever–basically everything in the search string (except perhaps the nonsense words like “and,” “at,” “by,” etc.), but what analyzer should I use? I’ve tried PatternAnalyzer and supplied .+ as the Pattern to look for but typing even just a single letter returns nothing.

Answer 1

I stuck with a StandardAnalyzer since there doesn’t seem to be an alternative. What I did is tokenize the string via a regex that captures non-word characters, combine them in an AND BooleanQuery, and combine the query for the two Fields in another OR BooleanQuery.

In my code below, entry is the word, description is the definition, and s is the search string as a CharSequence.

BooleanQuery bq = new BooleanQuery();
BooleanQuery entryBQ = new BooleanQuery();
BooleanQuery descriptionBQ = new BooleanQuery();
String[] tokens = String.valueOf(s).split("[^a-zA-Z0-9]");
for (String token : tokens) {
    if (token.isEmpty()) continue;
    entryBQ.add(new WildcardQuery(new Term("entry", token + "*")), BooleanClause.Occur.MUST);
    descriptionBQ.add(new WildcardQuery(new Term("description", token + "*")), BooleanClause.Occur.MUST);
}
bq.add(entryBQ, BooleanClause.Occur.SHOULD);
bq.add(descriptionBQ, BooleanClause.Occur.SHOULD);
TopScoreDocCollector collector = TopScoreDocCollector.create(10, true);
is.search(bq, collector);

This is horrendously slow in my Android app right now but I can optimize it later. 🙂