Question

Hello I have a 32mb file. It is a simple dictionary file encoded 1250 with 2.8 million lines in it. Every line has only one unique word:

cat
dog
god
...

I want to use Lucene to search for every anagram in dictionary of specific word. For example:

I want to search every anagram of the word dog and lucene should search my dictionary and return dog and god. In my webapp I have a Word Entity:

public class Word {
    private Long id;
    private String word;
    private String baseLetters;
    private String definition;
}

and baseLetters is the variable which are sorted letters alphabetically for searching such anagrams [god and dog words will have the same baseLetters: dgo]. I succeeded in searching such anagrams from my database using this baseLetters variable in different service but I have problem to create index of my dictionary file. I know I have to add to fields:

word and baseLetters but I have no idea how to do it 🙁 Could someone show me some directions to achieve this goal?

For now I have only something like that:

public class DictionaryIndexer {

private static final Logger logger = LoggerFactory.getLogger(DictionaryIndexer.class);

@Value("${dictionary.path}")
private String dictionaryPath;

@Value("${lucene.search.indexDir}")
private String indexPath;

public void createIndex() throws CorruptIndexException, LockObtainFailedException {
    try {
        IndexWriter indexWriter = getLuceneIndexer();
        createDocument();           
    } catch (IOException e) {
        logger.error(e.getMessage(), e);
    }       
 }

private IndexWriter getLuceneIndexer() throws CorruptIndexException, LockObtainFailedException, IOException {
    StandardAnalyzer analyzer = new StandardAnalyzer(Version.LUCENE_36);
    IndexWriterConfig indexWriterConfig = new IndexWriterConfig(Version.LUCENE_36, analyzer);
    indexWriterConfig.setOpenMode(OpenMode.CREATE_OR_APPEND);
    Directory directory = new SimpleFSDirectory(new File(indexPath));
    return new IndexWriter(directory, indexWriterConfig);
}

private void createDocument() throws FileNotFoundException {
    File sjp = new File(dictionaryPath);
    Reader reader = new FileReader(sjp);

    Document dictionary = new Document();
    dictionary.add(new Field("word", reader));
}

}

PS: One more question. If i register DocumentIndexer as a bean in Spring will the index be creating/appending every time I redeploy my webapp? and the same will be with the future DictionarySearcher?

Answer 1

The function createDocument() should be

private void createDocument() throws FileNotFoundException {
    File sjp = new File(dictionaryPath);
    BufferedReader reader = new BufferedReader(new FileReader(sjp));

    String readLine = null;
    while((readLine = reader.readLine() != null)) {
        readLine = readLine.trim();
        Document dictionary = new Document();
        dictionary.add(new Field("word", readLine));
        // toAnagram methods sorts the letters in the word. Also makes it
        // case insensitive.
        dictionary.add(new Field("anagram", toAnagram(readLine)));
        indexWriter.addDocument(dictionary);
    }
}

If you are using Lucene for a lot of functionality, considering using Apache Solr, a search platform built on top of Lucene.

You could also model your index with just one entry per anagram group.

{"anagram" : "scare", "words":["cares", "acres"]}
{"anagram" : "shoes", "words":["hoses"]}
{"anagram" : "spore", "words":["pores", "prose", "ropes"]}

This would require updates to existing documents in the index while processing your dictionary file. Solr would help with higher level API in such cases. For e.g., IndexWriter does not support updating documents. Solr supports updates.

Such an index would give one result document per anagram search.

Hope it helps.