Question

I need to save batch of objects.

Slow performance (150 000 objects are saved in 15 minutes) is due to checks.

Check for each object – the aggregate of some fields should be unique.

So the questions are:

• will it help if I create additional indexed column with hash-value of those fields and check that column before detailed check?
• is there any other solution?

Answer 1

Assuming you are currently checking row-by-row, and that table locking during the update is not a problem, I guess you’re trying to do a special case of UPSERT here (what happens if there IS already a matching row? Do you abort the process, or skip the row, or something else?)

In SQL Server 2008, a MERGE statement is probably what you’re looking for.

In SQL Server 2005 and earlier, you can use a LEFT JOIN to do something similar (where a key column in the joined table is null)

---

UPDATE(d twice): based on your feedback, let’s assume that you are using SQL Server 2005 and want to add new rows based on a compound (multi-column) key, while avoiding (and possibly alerting on) existing/duplicate rows.

You have a table “Table1” that contains some data:

KeyCol1  KeyCol2   ValueCol
1        2         Test
3        2         MoreTest
4        2         TestForever

You have a temp table (or table variable, or loading/staging table, or something) that contains the data you want to load:

KeyCol1  KeyCol2  ValueCol
2        2        OKValue
5        2        AlsoOK
4        2        ProblemValue

You do an inner join to get any “already existing” rows (and then raise an error, or do whatever you want with that list of problem entries):

SELECT FinalTable.* 
INTO #ProblemRecords
FROM FinalTable 
INNER JOIN TempTable ON FinalTable.KeyCol1 = TempTable.KeyCol1
    AND FinalTable.KeyCol2 = TempTable.KeyCol2
--Could add group by here if it's possible to have more than one record per join key

And a left join to insert the records that don’t have an already-matching record in the final table:

INSERT INTO FinalTable (KeyCol1, KeyCol2, ValueCol)
SELECT TempTable.KeyCol1, TempTable.KeyCol2, TempTable.ValueCol)
FROM TempTable
LEFT JOIN FinalTable ON FinalTable.KeyCol1 = TempTable.KeyCol1
    AND FinalTable.KeyCol2 = TempTable.KeyCol2
WHERE FinalTable.KeyCol1 Is Null

Some notes about this approach:

• For it to be safe, you would need to introduce large-scale locking (I have omitted transaction here for simplicity). This could be a problem for an OLTP system that needs to remain available and responsive.
• It’s going to use more resources than your “slowly but surely does it” approach, while it’s running.
• Somehow you need to get the data into a temporary/loading table to start with. Some sort of bulk-load process would probably make sense (see BULK INSERT, bcp, SQLXML, SSIS/Import-Export wizard, etc for this)