Question

Because I’m using a shared Microsoft SQL Server instance, I have started to use MongoDB as it’s faster to have around 10 inserts per second.

At the end of the day, I need to read from the MongoDB (MongoHQ) and insert all data into the shared SQL Server instance (hosting provider).

Currently I’m doing this flow:

1. Get 1000 rows from MongoDB
2. open a connection to SQL Server
3. For each row
   1. insert the data into SQL Server using a stored procedure (as calculations need to be done)
   2. update MongoDb row * (so we don’t pick up the processed row later)
4. close the connection to SQL Server
5. GO TO 1

And with this, I’m “wasting” around 3 seconds per row…

Is there a way to speed things up?

Currently using C# to perform all the code.

Answer 1

You have SQLbulkcopy class to your rescue.

Read more details here.

http://msdn.microsoft.com/en-us/library/system.data.sqlclient.sqlbulkcopy(v=vs.100).aspx

Thanks,
Naval