Question

I am currently thinking how to best store web-crawling results in a database. In another question document-oriented databases were recommended to use for a web-crawler project: Database for web crawler in python?

Now I am wondering if map/reduce is the right way for such classification and value-generation. At least it seems to be able to do such stuff (map for only classification like years or authors, and map/reduce for calculating numerical values which I cannot think of an example at the moment).

However, would map-reduce / DocumentStores also be able to give me the right documents for a given word? In a relational database I would have to use a JOIN on some tables and then get documents containing these words:

SELECT * FROM docs d 
JOIN doc_words dw ON dw.doc_id = d.id 
JOIN words w ON dw.word_id = w.id 
WHERE w.word = 'foo'

I guess DocumentStores are not capable of such an operation as they do not support fulltext index and are not intended to have many references / relations.

Would the better alternative be mixing several systems? E.g. one for searching by words, one for searching by different values if present (like year of publication, author, …)? I think DocumentStores are not so bad for storing the metadata, as sometimes there are specific values and sometimes not (and DocumentStores are easy to use across multiple servers if wanted, as soon as there are too many documents for one server). Yet, I am not sure what would the best way to implement searching for a collection of documents (including webpages, pdfs, images, which have always different meta-data, but often also need fulltext index).

To make a clear question: Should I use another database system together with DocumentStores, use DocumentStores alone (howto search for words quickly?) or another DB system alone?

PS: Another example for such a problem would be the linking between webpages, which cannot be saved in DocumentStores well either. However, OrientDB might solve this problem as it seems to combine graph database and document-oriented database.

Answer 1

Checkout RavenDB. It is a document DB with Map/Reduce queries, using Lucene under the hood, so full-text search is fully supported also within Map/Reduce queries.

Custom Lucene analyzers are supported as well, so there’s a lot of room for further full-text extensions.

Other features like Includes and Live Projections may give you everything else a simple Map/Reduce will be missing.