Question

I would like to create a c# console application to get about 1000000 rows of data.

add some filtering logic in code and generate xml feed.

The one I’m using is working fine but for 250K rows of the data, than I get out of memory exception.

Here is an example of code I use in web application I need to change it console application and make it efficient.

var xrFeed = new XmlTextWriter(File.Create(@"c:\Items.xml"), Encoding.UTF8);

xrFeed.WriteStartDocument();
xrFeed.WriteStartElement("Name");

IEnumerable<ItemClass> items = _source.GetItems();

if (items != null)
{ 
    foreach (var i in items)
    {                    
        xrFeed.WriteStartElement("ad");

            xrFeed.WriteStartElement("id");
            xrFeed.WriteCData(m.ListingId.ToString());
            xrFeed.WriteEndElement();

            xrFeed.WriteStartElement("firstParameter");
            xrFeed.WriteCData("parameter");
            xrFeed.WriteEndElement();

            xrFeed.WriteStartElement("secondParameter");
            xrFeed.WriteCData("parameter2");
            xrFeed.WriteEndElement();

            xrFeed.WriteStartElement("thirdParameter");
            xrFeed.WriteCData("parameter3");
            xrFeed.WriteEndElement();

        xrFeed.WriteEndElement();
    }

    xrFeed.WriteEndElement();
    xrFeed.WriteEndDocument();

    xrFeed.Flush();
    xrFeed.Close();

    Response.End();
    DataBind();         
}

Answer 1

Try flushing the writer every 1000 items or so. Also you might want to partially retrieve the data from your datasource.