Question

I am pulling 1M+ records from an API. The pull works ok, but I’m getting an out of memory exception when attempting to ReadToEnd into a string variable.

Here’s the code:

        XDocument xmlDoc = new XDocument();

        HttpWebRequest client = (HttpWebRequest)WebRequest.Create(uri);
        client.Timeout = 2100000;//35 minutes
        WebResponse apiResponse = client.GetResponse();

        Stream receivedStream = apiResponse.GetResponseStream();
        StreamReader reader = new StreamReader(receivedStream);

        string s = reader.ReadToEnd();

Stack trace:

at System.Text.StringBuilder.ToString()
at System.IO.StreamReader.ReadToEnd()
at MyApplication.DataBuilder.getDataFromAPICall(String uri) in
    c:\Users\RDESLONDE\Documents\Projects\MyApplication\MyApplication\DataBuilder.cs:line 578
at MyApplication.DataBuilder.GetDataFromAPIAsXDoc(String uri) in
c:\Users\RDESLONDE\Documents\Projects\MyApplication\MyApplication\DataBuilder.cs:line 543

What can I do to work around this?

Answer 1

XMLReader is the way to go when memory is an issue. It is also fastest.