Home/ Questions/Q 754431
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-14T15:00:45+00:00

I’m stuck in a huge problem where i need to handle huge data. I

  • 0

I’m stuck in a huge problem where i need to handle huge data.

I get a data set which has 5 datatables. One of the tables has around 3000 columns and 50,000 records.

I need to save this dataset completely in the SQL database as XML and i need to retrieve it later.

I CAN’T make any design changes to skip it. When i do dataset.GetXml() it returns a string of huge length which throws OutOfMemoryException. I knew that the string datatype has a certain limit to carry data.

But in sql table the xml column can hold upto 2 Gb. So how can i move this dataset back and forth between the database and my application?

Thank you

NLV

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Your design sounds flawed. Why do you need to retrieve the entire table into memory? Page through it instead – operate on the table in smaller batches – say, 1000 records at a time.

    You probably don’t even need to do that. Generally, database applications operate over just the data that’s changing. If your app needs to modify 10 rows, then retrieve just those ten rows. You’ll save not only memory, but all the time it takes to retrieve and post that data.

    3000 columns in the table? That also smells (really) bad. Look into normalizing your database.

    I know it feels easier to fix the immediate problem and move on. You’ll have many, many fewer problems in the long run, however, if you fix the (frankly quite severe) design problems up front.

    • 0