Question

I have a project to create a high traffic search engine similar to altavista.com. The windows .NET C# will be used for the project. I am looking for a good search engine database that can handle a very high load. I have taken a look at lucene and sql server 2008. I have read that lucene tends to get corrupt when the load is very high. So, I am considering sql server 2008 but I am uncertain sql server can handle a very high load over gigabytes of data.

The database will be given a chunk of data and needs to fetch similar data elsewhere in the table. it will act just the similar questions area on stackoverflow. It will need to search over millions of rows. Is sql server 2008 and lucene my only options

Answer 1

I have used Lucene sucessfully with high volume datas and i dint face data corruption problems.

But if you are worried about the standalone lucene, you can try apache solr instead, its a open-source search server based on the Lucene

Solr is the popular, blazing fast open
source enterprise search platform from
the Apache Lucene project. Its major
features include powerful full-text
search, hit highlighting, faceted
search, dynamic clustering, database
integration, and rich document (e.g.,
Word, PDF) handling. Solr is highly
scalable, providing distributed search
and index replication, and it powers
the search and navigation features of
many of the world’s largest internet
sites.

There is a .net client for solr in google code solr.net.. You can try this