Question

Recently, twitter engineer post a very impressed blog about using Lucence instead of mysql for their search architecture.
So, I’m curious about why they choose lucence and why does mysql does not meet their requirements? On the other hand, what’s the performance (or, scalability to say) bottleneck for the DBMS database system?

Any ideas are appreciated!

Thanks in Adv

Vance

Answer 1

Think about a Lucene index as something like the index you have in the back of some large reference books: for every important term that appears in the book, it lists all the pages in which it appears. So if you want to find all the places in the book where a term appears, you go to the index and get a list of pages.

What Lucene does is take documents, break them into their individual words (that process is called “tokenization”) then for each word/token write in its index that that word appears in that document.

Think of the index like a hashtable (it’s not really one, but it’s the same idea): the keys are the words/tokens and for each key there’s a bucket with a list references to documents (URIs, filenames) that contain that word. It doesn’t store the document itself – just a reference to it. When you do a search on Lucene you provide a keyword and get back the list of documents that contain that keyword which appear in its index.

MySQL and other RDBMS’s are optimized for storing and retrieving records – collections of pre-defined, ordered columns. When you place an index on a column it looks at the entire content of the column as a single unit. If that column is a piece of text, it does not break it down into words.