Question

The title for this one was quite tricky.

I’m trying to solve a scenario,
Imagine a survey was sent out to XXXXX amount of people, asking them what their favourite football club was.
From the response back, it’s obvious that while many are favourites of the same club, they all “expressed” it in different ways.
For example,

For Manchester United, some variations include…

• Man U
• Man Utd.
• Man Utd.
• Manchester U
• Manchester Utd

All are obviously the same club however, if using a simple technique, of just trying to get an extract string match, each would be a separate result.

Now, if we further complication the scenario, let’s say that because of the sheer volume of different clubs (eg. Man City, as M. City, Manchester City, etc), again plagued with this problem, its impossible to manually “enter” these variances and use that to create a custom filter such that converters all Man U -> Manchester United, Man Utd. > Manchester United, etc. But instead we want to automate this filter, to look for the most likely match and converter the data accordingly.

I’m trying to do this in Python (from a .cvs file) however welcome any pseudo answers that outline a good approach to solving this.

Edit: Some additional information
This isn’t working off a set list of clubs, the idea is to “cluster” the ones we have together.
The assumption is there are no spelling mistakes.
There is no assumed length of how many clubs
And the survey list is long. Long enough that it doesn’t warranty doing this manually (1000s of queries)

Answer 1

Google Refine does just this, but I’ll assume you want to roll your own.

Note, difflib is built into Python, and has lots of features (including eliminating junk elements). I’d start with that.

You probably don’t want to do it in a completely automated fashion. I’d do something like this:

# load corrections file, mapping user input -> output
# load survey
import difflib

possible_values = corrections.values()

for answer in survey:
    output = corrections.get(answer,None)
    if output = None:
        likely_outputs = difflib.get_close_matches(input,possible_values)
        output = get_user_to_select_output_or_add_new(likely_outputs)
        corrections[answer] = output
        possible_values.append(output)
save_corrections_as_csv