Question

I’m trying to implement a search function into my website, which is running on pyramid, and I was wondering what is the most efficient way of approaching this problem. I am currently looking into Whoosh and MySQL full text searching with SqlAlchemy. I need a fast and simple implementation, and wondering which one would be the best choice.

Answer 1

I tried using fulltext with the native database for a while and it just was too much work to keep things working across sqlite, mysql, and pgsql. I ported all the search code over to whoosh and have been really happy ever since. It performs well for small workloads, is pure python, and no server to setup.

You just implement it almost like writing and updating a file on disk. From what I’ve read it does well in the single millions of documents. I’m using it with some 18k documents with an index size of around 100MB. There’s a lot of flexibility to implement various tokenizing and other config with it. I really suggest people start there and if they out grow the whoosh, then look at starting up extra processes with elasticsearch, lucene/solr, and the like.

You can see how I’ve got it implemented here:

https://github.com/mitechie/Bookie/blob/develop/bookie/models/fulltext.py

and I update it using SqlAlchemy event hooks:

https://github.com/mitechie/Bookie/blob/develop/bookie/models/__init__.py#L663

and you can judge a basic implementation of it by searching:

https://bmark.us/search