Question

I want to replace any latin / accented characters with their basic alphabet letters and strip out everything that cant be converted

examples:

'ë' to be replaced with 'e'
'ß' to be replaced with 's' , 'ss' if possible, if neither then strip it

i am able to do this in c# code but im just not well experienced in MSSQL to solve this without taking many days

UPDATE: the data in the varchar column is populated from a trigger on another table which should have normal UNICODE text. i want to convert the text to ascii7 in a function to use for further processing.

UPDATE: i prefer a solution where this can be done in SQL only and avoiding custom character mapping. can this be done, or is it currently just not possible?

Answer 1

As Aaron said, I don’t think you can dispose of mapping tables entirely in SQL, but mapping characters to ASCII-7 should involve some fairly simple tables, used in conjunction with AI collations. Here there are two tables, one to map characters in the column, and one for the letter of the alphabet (which could be expanded if necessary).

By using the AI collations, I get around a lot of explicit mapping definitions.

-----------------------------------------------
-- One time mapping table setup
CREATE TABLE t4000(i INT PRIMARY KEY);
GO

INSERT INTO t4000 --Just a simple list of integers from 1 to 4000
SELECT ROW_NUMBER()OVER(ORDER BY a.x)
FROM (VALUES(1),(2),(3),(4),(5),(6),(7),(8),(9),(10)) a(x)
CROSS APPLY (VALUES(1),(2),(3),(4),(5),(6),(7),(8),(9),(10)) b(x)
CROSS APPLY (VALUES(1),(2),(3),(4),(5),(6),(7),(8),(9),(10)) c(x)
CROSS APPLY (VALUES(1),(2),(3),(4)) d(x)
GO

CREATE TABLE TargetChars(ch NVARCHAR(2) COLLATE Latin1_General_CS_AI PRIMARY KEY);
GO

INSERT TargetChars -- A-Z, a-z, ss
SELECT TOP(128) CHAR(i)
FROM t4000
WHERE i BETWEEN 65 AND 90
    OR i BETWEEN 97 AND 122
UNION ALL
SELECT 'ss'
-- plus any other special targets here
GO

-----------------------------------------------
-- function
CREATE FUNCTION dbo.TrToA7(@str NVARCHAR(4000))
RETURNS NVARCHAR(4000)
AS
BEGIN
    DECLARE @mapped NVARCHAR(4000) = '';
    SELECT TOP(LEN(@str))
    @mapped += ISNULL(tc.ch, SUBSTRING(@str, i, 1))
    FROM t4000
    LEFT JOIN TargetChars tc ON tc.ch = SUBSTRING(@str, i, 1)
    COLLATE Latin1_General_CS_AI;

    RETURN @mapped;
END
GO

Usage example:

SELECT dbo.TrToA7('It was not á tötal löß.');

Result:

--------------------------
It was not a total loss.