What’s the fastest way to get a thousand records into the database?
I’m not looking for the best answers – I’m just looking to hear that they’ve done some work to performance tune their queries. If they’re doing fully logged individual record inserts, one at a time, into a data warehouse-size system, we’re going to have problems down the road. (Yes, I’ve actually worked with a BI developer that did millions of individual inserts per night in full recovery mode and thought the performance was the database’s fault.)
In a very generic sense, the fastest process to do this often involves some form of bulk insert, maybe paying attention to the options that ignore referential integrity, constraints, triggers and such (if this makes sense for your app), and possibly done under a recovery model that doesn’t log the entire thing.
If you’re coming from your app, you might look at something like the .net SqlBulkCopy class rather than writing the file and such as in the above.
As always when trying to get the absolute best performing method, test multiple methods in your environment.
Please tell us what you are looking for, then.
Okay, I have worked for six years with a platform which relies upon inserting many thousands of records at a time, many hundreds of times per day, and be as close to appearing as a decently performing web-app as possible. The bulk insert was one of my first “fixes” when I started maintaining the thing, and we’ve done much since then to optimize. I’m sure others out there have much more extreme requirements and use similar methods. Not that any of that really matters to the answer.
UPDATE: a word about hardware:
I’m sure that anyone looking for the absolute best performance when repeatedly stuffing a ton of records into a database will want to look at getting some of the best hardware available as well. This discussion would be beyond the scope of this site, I believe.