Home/ Questions/Q 6241513
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-24T11:46:52+00:00

I have a set of 200M documents I need to index. Every document has

  • 0

I have a set of 200M documents I need to index. Every document has a free text and additional set of sparse metadata information (100+ columns).

It seems that the right tool for free text indexing is Lucene while the right tool for structured sparse metadata is HBase.

I would need to query the data and join between free text search results and the structured data results (e.g. get all books that has the phrase “good morning” in their textand were first published in 1980).

What tools/mechanism should I look at to join structured and unstrcutured queries?
Results may include millions of records (before and after the join)

Thanks
Saar

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    A couple of things come to mind, in addition to lucene on hbase:

    1) Solr/Lucene can store multiple fields, and each field can have different types. So your date range example is plausible wholly within Solr.

    2) If you are talking about truly huge data sets that require a cluster, also look at ElasticSearch: http://www.elasticsearch.org/

    3) Lily attempts to answer your exact question http://www.lilyproject.org/lily/index.html

    • 0