I’m currently writing a script tasked with going through tens of thousands of rows of account information and cleaning mistyped addresses, as well as printing out reports on how the address was cleaned. Currently the biggest source of unclean addresses is mistyped street-names (it’s amazing how many ways you can spell a street-name). In any case, currently my script grabs the input street-name and performs a series of edits specific to the Norwegian language (v. becomes vegen, gt. becomes gata etc.) and searches for the street-name in a ~2 million row database of addresses. If it doesn’t find a match it proceeds to split off the latter half of the street-name and replacing it with a wildcard. It tries out different variations of the wildcard search.
Anyway, my question is:
Does MySQL include anything that could make this easier for me? I recall hearing mention of a "search" function in MySQL that finds the cells in a column with the most matching characters or something. In the cases where my wild-card search fails it would be a great tool to have.
Anything else that would help with finding matches to mistyped addresses would be great.
One option might be to try to use SOUNDEX to get you close to what you want. SOUNDEX will make matches off of pronunciation so it might get you closer if people are mistyping based off of the phonetic spelling of a street name.
You might also try the Levenshtein distance algorithm. This is probably more closely tied to what you are looking for. Basically it looks at how close one word is to another. It can be used for spell checking, etc. and it might be useful when looking for bad data in address fields. Here is a link to it:
http://www.merriampark.com/ld.htm
If you want the function to use the Levenshtein distance algorithm in MySQL, you can look at an example here:
http://www.artfulsoftware.com/infotree/queries.php#552