Question

I’m doing some web scraping to build up a personal SQL database. As I’m looping through the web requests, I’m adding records. The only thing is, duplicates sometimes appear in the web requests and I want to make sure to only add a record if it doesn’t already exist in my database. I gather this can be done by performing an SQL query before every insert to make sure that record hasn’t already been added, but is this the best way to do it? Would it make more sense to build up a Generic.List first, and then do all my database inserts at the end?

Answer 1

You can create a stored procedure that will attempt to update a record and then insert if the update query did not update any rows. This will minimize the number of queries that need to be run and prevent checking for the row’s existence. A little bit of Googling found this. The second option looks like it might be what you are looking for.

/*
Same SP is used to INSERT as well as UPDATE a table.
Here we are avoid unnecessary checking of whether the record exists or not.
Instead try to Update directly. If there is no record then @@RowCount would be 0.
Based on that Insert it as a new record.
*/
CREATE PROCEDURE uspUPSert_Ver2
(
    @empID INT,
    @fname VARCHAR(25),
    @lname VARCHAR(25),
    @emailid VARCHAR(50)
)
AS
BEGIN
    SET NOCOUNT ON
    BEGIN TRAN
        UPDATE tblUpsert WITH (SERIALIZABLE)
            SET emailid = @emailid ,
                firstname = @fname ,
                lastname = @lname
        WHERE EmpID = @empID

        IF @@ROWCOUNT = 0
        BEGIN
            INSERT INTO tblUpsert VALUES (@empID, @fname, @lname, @emailid)
        END
    COMMIT TRAN
END
GO