Question

Currently one my DAL methods uses the ExecuteXmlReader command to read and append to a StringBuilder object and return XML as a string. I realize this isn’t the best way to handle very large files (200MB+) as when the ToString() method is called on StringBuilder it throws an out of memory exception.

There are certain constraints I need to adhere by – I cannot return a XMLReader out of the DAL (db connection stays open, etc), this XML will have to be written to file and keeping it’s formatting for various reasons, currently using an XMLTextWriter’s WriteRaw method for that purpose.

What would be the best data type to return from the DAL to make all this work with large XML files?
I explored the options of using a DataSet, Memory Stream and a Byte Array – but each have their shortfalls. I am looking for the best practice and some code samples would be helpful.

Answer 1

You can use XDocument the work with your xml files. You can use linq to xml to query through your elements. An example is below:

XDocument document = XDocument.Load("http://someurl.com/data.xml");

var elements = from element in document.Descendants("SomeElement")
               select element;

Console.WriteLine(elements.First().Value);

If you also get the out of memory exception with this (i think you will), then you should take a look at the following post:

http://james.newtonking.com/archive/2007/12/11/linq-to-xml-over-large-documents.aspx

More information about the XDocument class can be found here:

http://msdn.microsoft.com/en-us/library/system.xml.linq.xdocument.aspx

and here

http://www.hookedonlinq.com/Print.aspx?Page=LINQtoXML5MinuteOverview

Good luck!