Question

I need to handle very large datatables (2 million rows+) that comes from databases (SQL, Oracle, Access, MySQL, Sharepoint etc) outside of my control: Currently I loop through every row and column building a string object, but I run out of memory at about 100k rows.

The only solution I may take is to break the datatable into smaller pieces and persisting each block before starting on the next block of rows.

Since I cannot add ROW_NUMBER() or anything similar, I have to handle the populated datatable.

How can I easily (keep performance in mind) break the populated datatable into smaller datatables like paging?

PS there is no visual component to this functionality.

Answer 1

Are you using string concatenation? like this string += string.

Change that to StringBuilder and you should not have problems, at least not for 20k rows.