Question

I’m writing an import utility that is using phone numbers as a unique key within the import.

I need to check that the phone number does not already exist in my DB. The problem is that phone numbers in the DB could have things like dashes and parenthesis and possibly other things. I wrote a function to remove these things, the problem is that it is slow and with thousands of records in my DB and thousands of records to import at once, this process can be unacceptably slow. I’ve already made the phone number column an index.

I tried using the script from this post:
T-SQL trim &nbsp (and other non-alphanumeric characters)

But that didn’t speed it up any.

Is there a faster way to remove non-numeric characters? Something that can perform well when 10,000 to 100,000 records have to be compared.

Whatever is done needs to perform fast.

Update
Given what people responded with, I think I’m going to have to clean the fields before I run the import utility.

To answer the question of what I’m writing the import utility in, it is a C# app. I’m comparing BIGINT to BIGINT now, with no need to alter DB data and I’m still taking a performance hit with a very small set of data (about 2000 records).

Could comparing BIGINT to BIGINT be slowing things down?

I’ve optimized the code side of my app as much as I can (removed regexes, removed unneccessary DB calls). Although I can’t isolate SQL as the source of the problem anymore, I still feel like it is.

Answer 1

I may misunderstand, but you’ve got two sets of data to remove the strings from one for current data in the database and then a new set whenever you import.

For updating the existing records, I would just use SQL, that only has to happen once.

However, SQL isn’t optimized for this sort of operation, since you said you are writing an import utility, I would do those updates in the context of the import utility itself, not in SQL. This would be much better performance wise. What are you writing the utility in?

Also, I may be completely misunderstanding the process, so I apologize if off-base.

Edit:
For the initial update, if you are using SQL Server 2005, you could try a CLR function. Here’s a quick one using regex. Not sure how the performance would compare, I’ve never used this myself except for a quick test right now.

using System;   using System.Data;   using System.Text.RegularExpressions;   using System.Data.SqlClient;   using System.Data.SqlTypes;   using Microsoft.SqlServer.Server;    public partial class UserDefinedFunctions   {       [Microsoft.SqlServer.Server.SqlFunction]       public static SqlString StripNonNumeric(SqlString input)       {           Regex regEx = new Regex(@'\D');           return regEx.Replace(input.Value, '');       }   };   

After this is deployed, to update you could just use:

UPDATE table SET phoneNumber = dbo.StripNonNumeric(phoneNumber)