What would be considered best practice when working with the following scenario?
I have a dataset of ~6 million records total, broken down into 30+ tables with some tables having a few hundred thousand records. I need to use an api that only allows an insert of 200 records at a time.
I am breaking down the insert by table. Now here is where I am considering the first two options that came to my mind. I can either get the full dataset for that table and then in C# loop through the dataset only inserting 200 at a time. Or I can make multiple database calls grabbing 200 records at a time. Creating my object and making the call to the API to insert my records. Which I think I could make the next database call while the other records are being inserted.
Use multiple threads, with each reading and inserting 200 at a time. Not only will this be faster, but you won’t run into potential OutOfMemoryExceptions with that large of a dataset by being unable to allocate contiguous blocks over fragmented memory.