Question

I’m running a setup with django 1.4, Haystack 2 beta, and ElasticSearch .20. My database is postgresql 9.1, which has several million records. When I try to index all of my data with haystack/elasticsearch, the process times out and I get a message that just says “Killed”. So far I’ve noticed the following:

1. I do get the number of documents to get indexed, so I’m not getting an error like, “0 documents to index”.
2. Indexing a small set, for example 1000, works just fine.
3. I’ve tried hardcoding the timeout in haystack/backends/__init__.py and that seems to have no effect.
4. I’ve tried changing options in the elasticsearch.yml also to no avail.

If hardcoding the timeout doesn’t work, then how else can I extend the time for indexing? Is there another way to change this directly in ElasticSearch? Or perhaps some batch processing method?

Thanks in advance!

Answer 1

This version of haystack is buggy. The line of code causing the problem was found in the file haystack/management/commands/update_index.py in the following line:

pks_seen = set([smart_str(pk) for pk in qs.values_list('pk', flat=True)])

Is causing the server to run out of memory. However, for indexing, it does not seem to be needed. So, I just changed it to:

pks_seen = set([])

Now it’s running through the batches. Thank you everyone that answered!