Question

What is the best way to handle synonyms (phrases) using Lucene?
Especially, when I need to execute queries like :a OR b OR c NOT d

How about adding a new field called “synonyms” to each document while indexing?
This field’s value would have a list of all synonyms. It would be added to a document only when that document has any of the synonyms.

I would then execute an “OR” search query which would look for search keyword in this field along with other fields.

Can this approach work well for any kind of query?

FYI,
The synonyms in my application are totally custom and not from English dictionary…ie. “Global Leader in Finance” could also mean “Top Investment Bank” or “Fortune 500 Finance company” etc etc.

Please suggest.

Thanks.

Answer 1

There is a contribution to the Lucene project called “wordnet”. According to its documentation:

This package uses synonyms defined by WordNet to build a Lucene index storing them, which in turn can be used for query expansion. You normally run Syns2Index once to build the query index/”database”, and then call SynExpand.expand(…) to expand a query.

It includes a sample of what it does:

If you pass in the query “big dog” then it prints out:

Query: big adult^0.9 bad^0.9 bighearted^0.9 boastful^0.9 boastfully^0.9 bounteous^0.9 bountiful^0.9 braggy^0.9 crowing^0.9 freehanded^0.9 giving^0.9 grown^0.9 grownup^0.9 handsome^0.9 large^0.9 liberal^0.9 magnanimous^0.9 momentous^0.9 openhanded^0.9 prominent^0.9 swelled^0.9 vainglorious^0.9 vauntingly^0.9
dog andiron^0.9 blackguard^0.9 bounder^0.9 cad^0.9 chase^0.9 click^0.9 detent^0.9 dogtooth^0.9 firedog^0.9 frank^0.9 frankfurter^0.9 frump^0.9 heel^0.9 hotdog^0.9 hound^0.9 pawl^0.9 tag^0.9 tail^0.9 track^0.9 trail^0.9 weenie^0.9 wiener^0.9 wienerwurst^0.9

You see that the original words (“big” and “dog”) have no weighting attached to them. The synonyms, however, have a weighting (0.9) that you can configure yourself.

It comes bundled with the standard distribution of Lucene, in the “contrib” directory.