Question

I am using elasticsearch 0.20.2 on EC2 instance (16GB RAM, no swap enabled). I have a lot indexed documents and when I try to do facet results I got heap space error and elasticsearch server is unusable then. I was increasing heap memory for java but nothing helps. So my question is can I limit number of documents on which facet will be applied.

Here is my settings and mapping:

my_settings = {
    'settings': {
        'analysis': {
            'analyzer': {
                'text_analyzer': {
                    'tokenizer': 'standard',
                    'filter': ['standard', 'lowercase']
                },
                'suggestions_analyzer': {
                    'tokenizer': 'standard',
                    'filter': ['suggestions_shingle']
                }
            },
            'filter': {
                'suggestions_shingle': {
                    'type': 'shingle',
                    'min_shingle_size': 2,
                    'max_shingle_size': 5
                }
            }
        }
    }
}

my_mapping = {
    'test-type':{
        'properties':{
            'publish_datetime': {'type': 'date'},
            'text': {
                'type': 'multi_field',
                'fields': {
                    'text': {'type': 'string', 'analyzer': 'text_analyzer', 'include_in_all': True},
                    'suggestions': {'type': 'string', 'analyzer': 'suggestions_analyzer', 'include_in_all': False}
                }
            }
        }
    }
}

and my search query is:

query = {
    'filtered': {
        'filter' : {
            'limit' : {'value' : 10}
        },
        'query':{
            'prefix':{
                    'text.suggestions': 'wha'
            }
        },
    },
    'facets':{
        'text_suggestions':{
            'terms':{
                'field':'text.suggestions',
                'regex':'^%s.*' % 'wha',
                'size': 5
            }
        }
    },
    'size': 0
}

Is someone succeed to limit document number which will be facet let share with us.

Answer 1

No, it is not possible to artificially limit the number of documents on which to run the aggregation — that aggregation would clearly be incorrect. The facets are limited by the query you issue (and also by any facet_filters you use).

So, several ways to keep in mind:

• In general, facets will be limited by the query you send to Elasticsearch
• Values for fields you’re faceting on will be loaded into memory
• There are some techniques to prevent out of memory errors for ad-hoc faceting, but they are not useful for regular user-exposed searching/faceting

Please see the Estimating field cache size for facets in advance discussion on the mailing list.

Based on the query you have posted, it’s not clearly what are you trying to do with faceting on the text.suggestions field?