We have a small data set and would like to search through it in a clever way in order to populate an auto-complete input box. The user wants to search for one type of entity, but should be able to put in attributes of associated types.
As our application is very industry-specific, I will try to put the abstract question into a more relatable context:
Example: The user wants to select a certain type of fruit and can use a lot of different characteristics of the fruit to search for it. She may search for its name (apple), special types of the same fruit (golden delicious), localized versions (apfel, تفاح, pomme), but also other characteristics, such as food containing the fruit (pandowdy, cider), or even results from a full-text search of its description.
Normally I would want to use something like solr or elasticsearch, but our data set is not really large with 4-5 associated entities of max 200 elements each. In addition it will stay very static, with a few editorial additions every other week. So we thought about using the browser’s data store to manage and search some kind of index. The problem is that in our research we could not find anything that goes far beyond abstracting Web SQL or Indexed DB.
Are there any projects or libraries that do something like solr or elasticsearch and can be used on the client-side for small data-sets? We are looking for the following features
- Fast search
- Pre-processing (tokenization, filtering …)
- Ordering
- Ranking, query boosting
It should work on recent browsers and mobile, dumb fallback for other browsers (i.e. full text search) acceptable
IndexedDB
Is incredibly cumbersome to work with and I don’t think it will do what you want without some modifications.
WebSql
I use Chrome WebSql (sqlite) with fulltext (FT3) and you can do some advanced querying with it and it automatically weights results. eg:
table_ft3 structure=[id, name, description, pets]
and you can query with
“SELECT id FROM table_ft3 WHERE table_ft3 MATCH ‘word1 word2 wordfuzzy* -notme +required name:john edwar* mary pets:dog cat'”
This is, however, restricted to Chrome. On a table of 560k rows with about 12 words each it’s very fast, 10ms and less.
JSLinq – http://jslinq.codeplex.com/
This is good for large data sets, by storing everything in memory it is ridiculously quick, but I don’t think it has any advanced weighting. It does have the added benefit of allowing you to write your own complex query functions.
JSII – http://karussell.wordpress.com/2010/11/02/jsii-full-text-search-in-1k-loc-of-javascript/
This uses Lucene style weighting and on 20k rows it comes in at 50ms which is not great but probably good enough for your data.