I have a function that loops through my list of objects, makes some checks on them and saving to an SQL server DB.
The number of objects I should save can be tens of thousands, so I decided to use parallel framework to make it faster. I use parallel.foreach to do that.
This iteration is in a backgroundworker thread.
It works, but I recognized it is slower than ‘normal’ foreach. E.g. in my last test a list of my objects are processed in 2:41.35 minutes, regular foreach do the does it in 2:14.92. The difference is nearly half a minute.
Is it a common behavior for DB insert operations in parallel framework? Is it advised to use parallel.foreach to make DB insert operations?
Thx!
VS2010/.Net4/c#
If you have a single database connection, trying to save objects in parallel is probably just adding synchronization overhead on the client side (the connection object). See the potential pitfalls of parallelism. Without knowing anything about your particular code, I would guess that a better approach would be to try to parallelize the checking only, and then do the saving (using the single db connection) sequentially. Of course, if your objects are mutable you need to make sure they can’t change between checking and saving, but that’s still true in the current approach.
BTW, are you saving all of these objects to the db in a single transaction or do you just have autocommit on? It depends on what kind of integrity constraints apply to your insertions, of course, but 2-3 minutes seems like a long time even for tens of thousands of rows. Using transactions wisely, if you aren’t currently, is likely to get you a significant increase in performance.
Edited to add: Using SQLite on my desktop machine, I can insert 100 1-field rows in 5 seconds using autocommit. I can insert 10,000 1-field rows in 13 seconds if they’re in a single transaction.