I am about to deploy a web app using MongoDB as storage. The problem is, that locally I can build the indexes whenever I want (also done automatically) – but you shouldn’t do that in production. Now switching to production I am not quite sure on when to build the indexes and how often. Should I do it every day? Should I do it once a week? My project is like a small social network – so let’s say I will have about 100.000 users someday. Depending on the amount of data it could take hours or even days to build indexes.
I intend to build my indexes in the background (see here http://www.mongodb.org/display/DOCS/Indexing+as+a+Background+Operation). It will still cause a write lock, but the build job will pause to allow other readers and writers to access the database – so if I there would be a heavy load on the database, a background index build would still affect the performance in a negative way. So it should be done within a time window where the traffic is at a minimum (researched here http://www.javabeat.net/articles/353-indexing-in-practice-using-mongodb-1.html).
Does anybody have experience on when and how often to create indexes in production (preferably with a large amount of data)? I am using Rails with Mongoid (http://mongoid.org/).
You seem to assume you have to periodically rebuild indexes. This is not the case. MongoDB updates indexes as you write to the related data.
The only possible reason I can see for production index rebuilds are :