Question

We have 16 core on a server, each of them has 4.5 millions customer orders. Recently, we commit 200 new orders into each core per hour, then optimize all cores. We find that every optimize operation need 30min at least. I have several questions:

1. We should optimize after every commit? Would it decrease query performance significantly in our circumstance if we optimize per day?

2. We only add new orders into solr, will never update or delete any one in solr. So ,can we only optimize the index which we commit, in other word, can we optimize index by date range?

Answer 1

No, do not optimize after every commit. The frequency with which you should optimize is dependent on how often you update.

From http://wiki.apache.org/solr/SolrPerformanceFactors#Optimization_Considerations

If you have a rapidly changing index, rather than optimizing, you
likely simply want to use a lower merge factor. Optimizing is very
expensive, and if the index is constantly changing, the slight
performance boost will not last long. The tradeoff is not often worth
it for a non static index.

The question of “will it decrease query performance significantly” is something you’ll have to test for yourself, but measure, measure, measure. Then, decide if the performance is actually a problem. If a 50ms query becomes a 60ms query, that’s a 20% increase in response time, but does it matter? Only you can answer those tradeoffs. But you MUST have numbers.