I have a large amount of data in some files, the data cannot even be in memory. As of now I am parsing the data, and after each entity is parsed, the parsing class raises an event, in this event I am, using LINQ, inserting an item into the corresponding Database table. When a whole file (have also tried all files) is parsed, any inserts are submitted to the database. The problem is that this takes way too long. I have left the program running over night and it did not even finish. The data is about 1.5gb of data on disk. How can I speed up my insertions? I am leveraging parallelisation for parsing, and it takes not time at all to parse, it is the insertions that are creating a huge bottleneck.
Share
bulk insert helped, along with a pipeline model, and general code efficiency. It’s never going to take less than 30 minutes unless you’re on a fast machine with fast internet…