Question

I am newbie to Solr and have no skills in Java, so maybe I am missing something… I am trying to get Solr to strip off HTML from content, using the following CharFilter:

http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#solr.HTMLStripCharFilterFactory

This is how I am including it into my schema.xml:

<fieldType name="text" class="solr.TextField" positionIncrementGap="100">
  <analyzer type="index">
    <charFilter class="solr.HTMLStripCharFilterFactory"/>
    <tokenizer class="solr.StandardTokenizerFactory"/>
  </analyzer>
  <analyzer type="query">
    <charFilter class="solr.HTMLStripCharFilterFactory"/>
    <tokenizer class="solr.StandardTokenizerFactory"/>
  </analyzer>
</fieldType>

<fields>
  <field name="text" type="text" indexed="true" stored="true" multiValued="false" />
</fields>

If I query using the analyzer on the Solr admin panel for “d'Hèrcules”, I get a match (see field):

<doc>
<long name="comment_count">0</long>
<str name="ct_model_name">theatre</str>
<str name="django_ct">timeout.work</str>
<str name="django_id">2535</str>
<bool name="family">false</bool>
<long name="hits">0</long>
<str name="id">timeout.work.2535</str>
<str name="name">Les aventures d'Hèrcules</str>
<arr name="parent_sections">
<str>Escena</str>
</arr>
<long name="rating">0</long>
<bool name="recommended">false</bool>
<arr name="sections">
<str>Escena - Infantil</str>
</arr>
<str name="text">
Les aventures d&#39;Hèrcules Jordi Andújar &lt;p&gt;Prepareu-vos per viatjar a l’antiga Grècia on coneixereu l’heroi més gran de tots els temps: l’Hèrcules. De viatge cap a l’Olimp, l’heroi viurà les més increïbles aventures, lluitarà amb bèsties ferotges i perillosos monstres, i s’enfrontarà a la maldat de la temible deessa Hera. Per a tota la família&lt;/p&gt;
</str>
...
</doc>

but I need to match by searching the non-HTML-entity form: e.g. in this case, would be “l’Hèrcules” (note the single quote).

What am I doing wrong?

By the way, I am using django-haystack, if this info is useful in any way.

Thanks in advance,

Hector

Answer 1

You’re doing well on Solr side, the only thing you should notice is that there’s a difference between what Solr stores and what Solr indexes. Solr stores data as they are without modifying anything, while through the analyzer chain you can change the way Solr indexes data.
So you are actually telling Solr to strip any html tag from the index and to replace html entities with related characters, but Solr will always return them as you submitted them.

You can search for d'Hèrcules or d'Hèrcules and you should have the same results, since you have the HtmlStripCharFilterFactory fired even at query time, so you are basically submitting the same query for d'Hèrcules in both cases because ' is converted to '.

Your search for l'Hèrcules makes me think that the unescaped character between l and Hercules in your document isn’t ' but something that looks really similar. I think you should check this.