Home/ Questions/Q 3841112
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-19T15:36:33+00:00

I’m not asking about implementing the spell check algorithm itself. I have a database

  • 0

I’m not asking about implementing the spell check algorithm itself. I have a database that contains hundreds of thousands of records. What I am looking to do is checking a user input against a certain column in a table for all these records and return any matches with a certain hamming distance (again, this question’s not about determining hamming distance, etc.). The purpose, of course, is to create a “did you mean” feature, where a user searches a name, and if no direct matches are found in the database, a list of possible matches are returned.

I’m trying to come up with a way to do all of these checks in the most reasonable runtime possible. How can I check a user’s input against all of these records in the most efficient way possible?

The feature is currently implemented, but the runtime is exceedingly slow. The way it works now is it loads all records from a user-specified table (or tables) into memory and then performs the check.

For what it’s worth, I’m using NHibernate for data access.

I would appreciate any feedback on how I can do this or what my options are.

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Calculating Levenshtein distance doesn’t have to be as costly as you might think. The code in the Norvig article can be thought of as psuedocode to help the reader understand the algorithm. A much more efficient implementation (in my case, approx 300 times faster on a 20,000 term data set) is to walk a trie. The performance difference is mostly attributed to removing the need to allocate millions of strings in order to do dictionary lookups, spending much less time in the GC, and you also get better locality of reference so have fewer CPU cache misses. With this approach I am able to do lookups in around 2ms on my web server. An added bonus is the ability to return all results that start with the provided string easily.

    The downside is that creating the trie is slow (can take a second or so), so if the source data changes regularly then you need to decide whether to rebuild the whole thing or apply deltas. At any rate, you want to reuse the structure as much as possible once it’s built.

    • 0