Question

I have the following Tire mappings on a Rails model:

   mapping do

     indexes :name, analyzer: 'arabic', boost: 10

     indexes :city_name, analyzer: 'arabic', boost: 5

     indexes :description, analyzer: 'arabic' 

   end

What is the best way to be able to have multiple languages covered (if I want to add English or something else)?

Answer 1

What matters is that you will need to index data using specific fields for each language.
Each field on which you want to apply language depending text analysis (stemming, stopwords, synonyms and so on) will need a specific mapping depending on the language itself. Let’s say you have a title and a content field, you are going to need a couple of those fields for each language.
Then you can decide to index all your documents on the same index, same type, and adding a field containing the language. Otherwise you can index each language as a different type, which would be underneath the same as the first option, but you wouldn’t need to manually specify the language filter at query time. The filter would be “automatically” applied as long as you specify the language as type within the url like this:

curl -X GET http://localhost:9200/index/en/_search -d '{
  "query" : {"match_all" : ""}
}

Otherwise you could also use a different index for each language. It really depends on your data, how big the index it’s going to be (how many documents per language?) and how you are going to make queries (always by language or can you mix different languages for instance?).

Also, the Data desing patterns talk that Shay recently gave at Berlin Buzzwords might be interesting for you.