I need to fill two DataTable from two different databases for some comparisons and read-only operations. That takes me to "select *" into these two DataTable.
However, the table I need to retrieve contains about 3 million rows in both databases. I get an OutOfMemoryExceptionwhile filling.
Is there a possibility to have these DataTable on disk rather than in memory?
Do you see any other solution to compare/perform read-only on such two huge sets? The things that I need to do make it very complicated to proceed by batches of subsets of rows.
My guess is that the ideal/efficient way of performing this would be to do it within the database. Unfortunately, your two tables are in two separate databases, which may cause complications.
Are these tables in two different databases on the same server (as in SQL Server instance), or are they in two separate servers? If they are within the same server/instance, then you could probably perform your comparison logic on the database server.
Otherwise, if these tables are on separate servers, then is there any possibility of setting up some kind of replication between databases so that you can get both data sets into a single database where you can perform your comparisons?