Question

I am currently using what I (mistakenly) thought would be a fairly straightforward implementation of Solr’s NGramTokenizerFactory, but I’m getting strange results that are inconsistent between the admin analyzer and actual query results, and I’m hoping for some guidance.

I am trying to get user inputs to match my NGram (minGramSize=2, maxGramSize=2) index. My schema for indexing and query time is below, in which

1. I strip all non alphanumeric characters using PatternReplaceCharFilter.
2. I tokenize with NGramTokenizerFactory.
3. I lowercase using LowerCaseFilterFactory (which leaves non-letter tokens in place, so my numbers will remain).

Using the schema below, I would think that a search for “PCB-1260” (with a properly escaped dash) should match an indexed Ngram tokenized and lowercased value of “Arochlor-1260” (i.e., the bigrams for 1260 are “12 26 60” in both the indexed value and the queried value).

Unfortunately, I get no results unless I delete the dash. [EDIT – even when I properly escape the dash and leave it in the query, I also get no results]. This seems odd because I’m doing a complete pattern replacement of all alphanumeric characters using PatternReplaceCharFilter – which I assume removes all whitespace and dashes.

The query analyzer in the admin page shows proper matching using the schema below – so I’m at a bit of a loss. Is there something fundamental about the PatternReplaceCharFilter or the NGramTokenizerFactory that I’m missing here?

I’ve checked the code and other posts, but can’t seem to figure this one out. After a week of banging my head against the wall, I submit this one to the authority of the stack….

<fieldtype name="tokentext" class="solr.TextField" positionincrementgap="100">
    <analyzer type="index">
        <charfilter class="solr.PatternReplaceCharFilterFactory" pattern="([^A-Za-z0-9])" replacement=""/>
        <tokenizer class="solr.NGramTokenizerFactory" mingramsize="2" maxgramsize="2"/>
        <filter class="solr.LowerCaseFilterFactory"/>
    </analyzer>
    <analyzer type="query">
        <charfilter class="solr.PatternReplaceCharFilterFactory" pattern="[^A-Za-z0-9]" replacement=""/>
        <tokenizer class="solr.NGramTokenizerFactory" mingramsize="2" maxgramsize="2"/>
        <filter class="solr.LowerCaseFilterFactory"/>
    </analyzer>
</fieldtype>

Answer 1

So – something is definitely odd with PatternReplaceCharFilter failing to remove dashes at query time. Ultimately, I just did some pre-query processing in php of the user input with preg_replace before sending to Solr, and – viola! – worked like a charm with the expected results. Puzzling that the PatternReplaceCharFilter wasn’t behaving…

Here’s the pre-query php code that I used to get rid of the dashes, if anyone needs it.

$pattern = '/([-])/';
$replacement = ' ';
$usrpar = preg_replace($pattern, $replacement, $raw_user_search_contents);
$res = htmlentities($usrpar, ENT_QUOTES, 'utf-8');

After that, I just passed $res to Solr…