Home/ Questions/Q 6088813
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-23T12:01:45+00:00

I am using hibernate-search-3.2.1.Final and would like to parse my input into shingles. From

  • 0

I am using hibernate-search-3.2.1.Final and would like to parse my input into shingles. From what i can see in the documentation, ShingleAnalyzerWrapper seem to be exactly what I needed. I have tested with both WhitespaceAnalyzer, StandardAnalyzer, and SnowballAnalyzer as the default analyzer for the ShingleAnalyzerWrapper.

Version luceneVersion = Version.LUCENE_29;
SnowballAnalyzer keywordAnalyzer= new SnowballAnalyzer(luceneVersion, "English", StopAnalyzer.ENGLISH_STOP_WORDS_SET);
ShingleAnalyzerWrapper shingleAnalyzer = new ShingleAnalyzerWrapper(keywordAnalyzer, 4);
shingleAnalyzer.setOutputUnigrams(false);
QueryParser keywordParser = new QueryParser(luceneVersion, "keyword", keywordAnalyzer);
Query keywordQuery = keywordParser.parse(QueryParser.escape(keyword.toLowerCase()));

However, the query came back empty. I was expecting keyword like “hello world, this is Lucene” to result in shingles [hello world this is, world this is lucene, this is lucene]

Let me know if my expectation and usage of ShingleAnalyzerWrapper is correct.

Thanks,
Ryan

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Maybe it’s a copy/paste error, but in your code snippet, the shingleAnalyzer is not actually being used because you’re passing the variable keywordAnalyzer to the query parser. What analyzer are you using at indexing time?

    If you use an analyzer that filters out stop words as the delegate analyzer for ShingleAnalyzerWrapper, stop words (“this” and “is” in your example) will be dropped before the shingle analyzer has a chance to create shingles from them.

    A good way to debug analyzers is to use something like AnalyzerUtils described in “Lucene in Action”. You can get the sample code here: http://java.codefetch.com/example/in/LuceneInAction/src/lia/analysis/AnalyzerUtils.java

    Nikita

    • 0