Question

We have a SQL Server table containing Company Name, Address, and Contact name (among others).

We regularly receive data files from outside sources that require us to match up against this table. Unfortunately, the data is slightly different since it is coming from a completely different system. For example, we have ‘123 E. Main St.’ and we receive ‘123 East Main Street’. Another example, we have ‘Acme, LLC’ and the file contains ‘Acme Inc.’. Another is, we have ‘Ed Smith’ and they have ‘Edward Smith’

We have a legacy system that utilizes some rather intricate and CPU intensive methods for handling these matches. Some involve pure SQL and others involve VBA code in an Access database. The current system is good but not perfect and is cumbersome and difficult to maintain

The management here wants to expand its use. The developers who will inherit the support of the system want to replace it with a more agile solution that requires less maintenance.

Is there a commonly accepted way for dealing with this kind of data matching?

Answer 1

Here’s something I wrote for a nearly identical stack (we needed to standardize the manufacturer names for hardware and there were all sorts of variations). This is client side though (VB.Net to be exact) — and use the Levenshtein distance algorithm (modified for better results):

    Public Shared Function FindMostSimilarString(ByVal toFind As String, ByVal ParamArray stringList() As String) As String         Dim bestMatch As String = ''         Dim bestDistance As Integer = 1000 'Almost anything should be better than that!          For Each matchCandidate As String In stringList             Dim candidateDistance As Integer = LevenshteinDistance(toFind, matchCandidate)             If candidateDistance < bestDistance Then                 bestMatch = matchCandidate                 bestDistance = candidateDistance             End If         Next          Return bestMatch     End Function      'This will be used to determine how similar strings are.  Modified from the link below...     'Fxn from: http://ca0v.terapad.com/index.cfm?fa=contentNews.newsDetails&newsID=37030&from=list     Public Shared Function LevenshteinDistance(ByVal s As String, ByVal t As String) As Integer         Dim sLength As Integer = s.Length ' length of s         Dim tLength As Integer = t.Length ' length of t         Dim lvCost As Integer ' cost         Dim lvDistance As Integer = 0         Dim zeroCostCount As Integer = 0          Try             ' Step 1             If tLength = 0 Then                 Return sLength             ElseIf sLength = 0 Then                 Return tLength             End If              Dim lvMatrixSize As Integer = (1 + sLength) * (1 + tLength)             Dim poBuffer() As Integer = New Integer(0 To lvMatrixSize - 1) {}              ' fill first row             For lvIndex As Integer = 0 To sLength                 poBuffer(lvIndex) = lvIndex             Next              'fill first column             For lvIndex As Integer = 1 To tLength                 poBuffer(lvIndex * (sLength + 1)) = lvIndex             Next              For lvRowIndex As Integer = 0 To sLength - 1                 Dim s_i As Char = s(lvRowIndex)                 For lvColIndex As Integer = 0 To tLength - 1                     If s_i = t(lvColIndex) Then                         lvCost = 0                         zeroCostCount += 1                     Else                         lvCost = 1                     End If                     ' Step 6                     Dim lvTopLeftIndex As Integer = lvColIndex * (sLength + 1) + lvRowIndex                     Dim lvTopLeft As Integer = poBuffer(lvTopLeftIndex)                     Dim lvTop As Integer = poBuffer(lvTopLeftIndex + 1)                     Dim lvLeft As Integer = poBuffer(lvTopLeftIndex + (sLength + 1))                     lvDistance = Math.Min(lvTopLeft + lvCost, Math.Min(lvLeft, lvTop) + 1)                     poBuffer(lvTopLeftIndex + sLength + 2) = lvDistance                 Next             Next         Catch ex As ThreadAbortException             Err.Clear()         Catch ex As Exception             WriteDebugMessage(Application.StartupPath , [Assembly].GetExecutingAssembly().GetName.Name.ToString, MethodBase.GetCurrentMethod.Name, Err)         End Try          Return lvDistance - zeroCostCount     End Function