Question

I have a table with about 100 columns and about 10000 rows.
Periodically, I will receive an Excel with similar data and I now need to update the table.
If new rows exist in Excel, I have to add them to the db.
If old rows have been updated, I need to update the rows in the db.
If some rows have been deleted, I need to delete the row from my main table and add to another table.

I have thought about proceeding as follows:

Fetch all rows from db into a DataSet.
Import all rows from Excel into a DataSet.
Compare these 2 DataSets now using joins and perform the required operations.

I have never worked with data of this magnitude and am worried about the performance. Let me know the ideal way to go about realizing this requirement.

Thanks in advance. 🙂

Answer 1

As my experience says, its so simple if you choose to do the stuff in t-sql as following:

1. You can use "OPENROWSET", "OPENQUERY", linked servers, DTS and many other thing in SQL Server to import the excel file into a temporary table.
2. You can write some simple queries to do that. If you are using SQL 2008, "MERGE" has exacly made for your question.

Another thing is that the performance is far different than C#. You can use "TOP" clause to chunk the comparison and do many other things.

Hope it helps.
Cheers