Question

I am working on the high-level design of a web application with the following characteristics:

• Millions of records
• Heavily indexed/searchable by various criteria
• Variable document schema
• Regular updates in blocks of 10K – 200K records at a time
• Data needs to remain highly available during updates
• Must scale horizontally effectively

Today, this application exists in MySQL and we suffer from a few huge problems, particularly that it is challenging to adapt to flexible schema, and that large bulk updates lock the data for 10 – 15 seconds at a time, which is unacceptable. Some of these things can be tackled by better database design within the context of MySQL, however, I am looking for a better “next generation” solution.

I have never used MongoDB, but its feature set seemed to most closely match what I am looking for, so that was my first area of interest. It has some things I am excited about, such as data sharding, the ability to find-update-return in a single statement, and of course the schema flexibility of NoSQL.

There are two things I am not sure about, though, with MongoDB:

1. I can’t seem to find solid
   information about the concurrency of
   updates with large data sets (see my
   use case above) so I have a hard
   time understanding how it might
   perform.

2. I do need open text search

That second requirement brought me to Lucene (or possibly to Solr if I kept it external) as a search store. I did read a few cases where Lucene was being used in place of a NoSQL database like MongoDB entirely, which made me wonder if I am over-complicating things by trying to use both in a single app — perhaps I should just store everything directly in Lucene and run it like that?

Given the requirements above, does it seem like a combination of MongoDB and Lucene would make this work effectively? If not, might it be better to attempt to tackle it entirely in Lucene?

Answer 1

Currently with MongoDB, updates are locking at the server-level. There are a few JIRAs open that address this, planned for v1.9-2.0. I believe the current plan is to yield writes to allow reads to perform better.

With that said, there are plenty of great ways to scale MongoDB for super high concurrency – many of which are simiar for MySQL. One such example is to use RAID 10. Another is to use master-slave where you write to master and read from slave.

You also need to consider if your “written” data needs to be 1) durable and 2) accessible via slaves immediately. The mongodb drivers allow you to specify if you want the data to be written to disk immediately (or hang in memory for the next fsync) and allow you to specify how many slaves the data should be written to. Both of these will slow down MongoDB writing, which as noted above can affect read performance.

MongoDB also does not have nearly the capability for full-text search that Solr\Lucene have and you will likely want to use both together. I am currently using both Solr and MongoDB together and am happy with it.