I notice strange behavior when running the the rebuild_index command multiple times from one Python shell session.
In this example I ran rebuild_index from the Bash shell and then entered the Python shell to check the facet values. They are capitalized. Then I ran the rebuild_index command from the Python shell and the facet values were in lowercase.
$ bin/manage rebuild_index
WARNING: This will irreparably remove EVERYTHING from your search index in connection 'default'.
Your choices after this are to restore from backups or rebuild via the `rebuild_index` command.
Are you sure you wish to continue? [y/N] y
Removing all documents from your index because you said so.
Failed to clear Elasticsearch index: Non-OK status code returned (404) containing u'IndexMissingException[[my_index] missing]'.
All documents removed.
Indexing 40 components.
In [1]: from haystack.query import SearchQuerySet
In [2]: SearchQuerySet().models(Component).facet('primary_language').facet_counts()
Out[2]:
{'dates': {},
'fields': {'primary_language': [(u'Java', 24),
(u'Scala', 5),
(u'C', 3),
(u'Python', 2),
(u'C++', 2),
(u'PHP', 1),
(u'Javascript', 1),
(u'Ada', 1),
(u'.Net', 1)]},
'queries': {}}
In [3]: from django.core import management
In [4]: management.call_command('rebuild_index', interactive=False, verbosity=0)
Failed to clear Elasticsearch index: Non-OK status code returned (404) containing u'IndexMissingException[[my_index] missing]'.
In [5]: SearchQuerySet().models(Component).facet('primary_language').facet_counts()
Out[5]:
{'dates': {},
'fields': {'primary_language': [(u'java', 24),
(u'scala', 5),
(u'c', 5),
(u'python', 2),
(u'php', 1),
(u'net', 1),
(u'javascript', 1),
(u'ada', 1)]},
'queries': {}}
The next weird thing is that if I run the rebuild_index command from the Python shell and try to order by the name field, I get an ElasticSearch error.
$ bin/manage rebuild_index
WARNING: This will irreparably remove EVERYTHING from your search index in connection 'default'.
Your choices after this are to restore from backups or rebuild via the `rebuild_index` command.
Are you sure you wish to continue? [y/N] y
Removing all documents from your index because you said so.
Failed to clear Elasticsearch index: Non-OK status code returned (404) containing u'IndexMissingException[[my_index] missing]'.
All documents removed.
Indexing 40 components.
In [1]: from haystack.query import SearchQuerySet
In [2]: SearchQuerySet().order_by('name')
Out[2]: [<SearchResult: my_app.component (pk=u'2')>, <SearchResult: my_app.component (pk=u'1')>, <SearchResult: my_app.component (pk=u'5')>, <SearchResult: my_app.component (pk=u'4')>, <SearchResult: my_app.component (pk=u'3')>, <SearchResult: my_app.component (pk=u'6')>, <SearchResult: my_app.component (pk=u'7')>, <SearchResult: my_app.component (pk=u'8')>, <SearchResult: my_app.component (pk=u'9')>, <SearchResult: my_app.component (pk=u'11')>, <SearchResult: my_app.component (pk=u'14')>, <SearchResult: my_app.component (pk=u'13')>, <SearchResult: my_app.component (pk=u'15')>, <SearchResult: my_app.component (pk=u'19')>, <SearchResult: my_app.component (pk=u'20')>, <SearchResult: my_app.component (pk=u'21')>, <SearchResult: my_app.component (pk=u'22')>, <SearchResult: my_app.component (pk=u'23')>, <SearchResult: my_app.component (pk=u'24')>, '...(remaining elements truncated)...']
In [3]: from django.core import management
In [4]: management.call_command('rebuild_index', interactive=False, verbosity=0)
Failed to clear Elasticsearch index: Non-OK status code returned (404) containing u'IndexMissingException[[my_index] missing]'.
In [5]: SearchQuerySet().order_by('name')
Failed to query Elasticsearch using '*:*': Non-OK status code returned (500) containing u'SearchPhaseExecutionException[Failed to execute phase [query], total failure; shardFailures {[SkgxFvdjRgyoiHv2qSpfbQ][my_index][4]: QueryPhaseExecutionException[[my_index][4]: query[filtered(ConstantScore(NotDeleted(cache(QueryWrapperFilter(django_ct:my_app.component)))))->cache(_type:modelresult)],from[0],size[20],sort[<custom:"name": org.elasticsearch.index.field.data.strings.StringFieldDataType$1@3cb9b947>]: Query Failed [Failed to execute main query]]; nested: IOException[Can\'t sort on string types with more than one value per doc, or more than one token per field]; }{[SkgxFvdjRgyoiHv2qSpfbQ][my_index][0]: QueryPhaseExecutionException[[my_index][0]: query[filtered(ConstantScore(NotDeleted(cache(QueryWrapperFilter(django_ct:my_app.component)))))->cache(_type:modelresult)],from[0],size[20],sort[<custom:"name": org.elasticsearch.index.field.data.strings.StringFieldDataType$1@249cf580>]: Query Failed [Failed to execute main query]]; nested: IOException[Can\'t sort on string types with more than one value per doc, or more than one token per field]; }{[SkgxFvdjRgyoiHv2qSpfbQ][my_index][3]: QueryPhaseExecutionException[[my_index][3]: query[filtered(ConstantScore(NotDeleted(cache(QueryWrapperFilter(django_ct:my_app.component)))))->cache(_type:modelresult)],from[0],size[20],sort[<custom:"name": org.elasticsearch.index.field.data.strings.StringFieldDataType$1@4ff80724>]: Query Failed [Failed to execute main query]]; nested: IOException[Can\'t sort on string types with more than one value per doc, or more than one token per field]; }{[SkgxFvdjRgyoiHv2qSpfbQ][my_index][2]: QueryPhaseExecutionException[[my_index][2]: query[filtered(ConstantScore(NotDeleted(cache(QueryWrapperFilter(django_ct:my_app.component)))))->cache(_type:modelresult)],from[0],size[20],sort[<custom:"name": org.elasticsearch.index.field.data.strings.StringFieldDataType$1@99b8b51>]: Query Failed [Failed to execute main query]]; nested: IOException[Can\'t sort on string types with more than one value per doc, or more than one token per field]; }{[SkgxFvdjRgyoiHv2qSpfbQ][my_index][1]: QueryPhaseExecutionException[[my_index][1]: query[filtered(ConstantScore(NotDeleted(cache(QueryWrapperFilter(django_ct:my_app.component)))))->cache(_type:modelresult)],from[0],size[20],sort[<custom:"name": org.elasticsearch.index.field.data.strings.StringFieldDataType$1@42d777c4>]: Query Failed [Failed to execute main query]]; nested: IOException[Can\'t sort on string types with more than one value per doc, or more than one token per field]; }]'.
Out[5]: []
After this I exited the Python shell. I started another Python shell. I ran the following lines and the result was correct.
from django.core import management
management.call_command('rebuild_index', interactive=False, verbosity=0)
from haystack.query import SearchQuerySet
SearchQuerySet().order_by('name')
This led me to the conclusion that rebuild_index can be run successfully only once during the life of a Python interpreter. It looks like running it a second time in a Python interpreter makes Haystack tokenize the values again. This explains why the facet values were returned in lowercase. It also explains the error message I got when trying to sort the results: “Can’t sort on string types with more than one value per doc, or more than one token per field”. It must be breaking the name values by the space character into multiple values.
Now here’s the question, why does it matter to Haystack if I run rebuild_index more than once in a Python interpreter? I don’t understand this because rebuild_index also runs clear_index as the first operation. Shouldn’t clearing the index be a sufficient way to ensure that rebuilding the index again gets you the same thing? Apparently this is not true. I have no idea why.
The real dilemma here is how on earth am I supposed to write proper unit tests (where rebuild_index is run in setUp()) if I can’t
1) retrieve facet values in their original formatting?
2) I can’t order by a field that has values with spaces?
Any explanations, suggestions, or ideas?
This happens because instance variables in the backend need to be reset before running
rebuild_indexa second time.