Question

In my application I have a feature where users can connect to different social networks to get a list of friends. For example, you can get your friends from linkedIn and from Facebook.

I would like to present a single list of friends that is the result of combining the various lists from different social networks together. The question is how to determine if names in different list are probably the same person? For example, Facebook might say “Jim Smith” and Linked In might say “Jim Smith, Phd” and I want my app to detected that they are the same person.

I have looked at the Levenshtein Distance function for strings, but I am not sure what to set the threshold too before considering two names as probably the same with a 75% confidence.

Here is what I am thinking about doing:

• Do all comparisons in lower case
• Remove all white space from the two names being compared before computing the levenshtein distance
• turn the levenshtien distance into a percentage of the length of the shorter name
• if the percentage is 0 its a perfect match
• if the percentage is < x they are probably the same

I am planning to use the apache commons StringUtils.getLevenshteinDistance() for the Leventstien computation.

What is a good value of x? 10%, 20%, 30% … etc? Is this a good algorithm my Math skills are pretty rusty and I am not sure if this will work.

Is there a better approach? Is there a standard library that one should use for something like this?

Answer 1

I would go for an automatic algorithm that decides what should be the threshold:

1. Create (manually) a set of examples – some are equivalent and some are not.
2. Run your algorithm with various thresholds.
3. Chose the threshold that maximize your result. I’d use the F-Measure, which takes into consideration the precision (how many that you said are “equivalent” are indeed are) and recall (how many of those who are correct are labeled as such).
4. Use statistical tools to determine if there is statistical significance between the different thresholds (it will help you know if you need more examples or your set is just fine). Wilcoxon test is the de-facto standard for it in most cases.

---

An alternative you might want to consider is in the field of machine learning – classification algorithms. In here, you want to calssify (user1,user2) and the answer is true if user1 is the same as user2.

You can use the same tools (statsitical tests, and using cross-validation) to estimate accuracy of this approach.

(Disclaimer: though I consider myself as experienced ML developer, I never tried to do something like this with this approach).