Question

I have created a couple of classes mean to represent a relational data structure ( parent child structures ). Below is an example of XML representation so far giving you an idea of what I mean

<BillingFile>
    <Account>
      <acctnum>122344231414</acctnum>
      <adjustments>34.44</adjustments>
      <Charges>
        <lineitem>
          <chargetype>PENALTY</chargetype>
          <amount>40.50</amount>
          <ratecode>E101</ratecode>
        </lineitem>
        <lineitem>
          <chargetype>LATE CHARGE</chargetype>
          <amount>445.35</amount>
          <ratecode>D101</ratecode>
        </lineitem>
      </Charges>
    </Account>
</BillingFile>

What I’m doing with my application is parsing through a large text file which could have upwards of 50,000+ accounts in it. Each time an account is read, I will create an “Account” object that has the parent objects, etc. The end goal is to be able to create an XML file containing all this account info that is serialized from the objects created.

The problem I see with this, is that if I store all these objects in memory it will cause a performance issue as it runs in those 50k+ record files.

What I’m wondering is, is there a way to sequentially serialize an object in C#, rather than all at once?

I’ve done some googling and it seems that the built in serialization methods of .NET are a one and done kind of deal. Is there a better way I can do this?

I’d rather avoid having to do any intermediate steps like storing the data in a database, since it’s easier to modify code than it is to mess with a bunch of tables and JOIN statements.

Thoughts?

Answer 1

XmlSerializer.Deserialize takes an XmlReader parameter. You could place the XmlReader just at the <Account> tag, and call the XmlSerializer there.

public IEnumerable<Account> ReadAccounts(TextReader source)
{
    var ser = new XmlSerializer(typeof(Account));

    using (var reader = XmlReader.Create(source))
    {
        if (!reader.IsStartElement("BillingFile"))
        {
            yield break;
        }

        reader.Read();

        while (reader.MoveToContent() == XmlNodeType.Element)
        {
            yield return (Account) ser.Deserialize(reader);
        }
    }
}

Similarly for serialization

public void WriteAccounts(IEnumerable<Account> data, TextWriter target)
{
    // Use XmlSerializerNamespaces to supress xmlns:xsi and xmlns:xsd
    var namespaces = new XmlSerializerNamespaces();
    namespaces.Add("", "");

    var ser = new XmlSerializer(typeof(Account));

    using (var writer = XmlWriter.Create(target))
    {
        writer.WriteStartElement("BillingFile");

        foreach (var acct in data)
        {
            ser.Serialize(writer, acct, namespaces);
            writer.Flush();
        }

        writer.WriteEndElement();
    }
}

You could also create a BillingFile class that implements IXmlSerializable, and put this functionality there.

Or if you prefer a push-based model instead:

public class AccountWriter : IDisposable
{
    private XmlWriter _writer;
    private XmlSerializer _ser;
    private XmlSerializerNamespaces _namespaces;

    private bool _wroteHeader = false;
    private bool _disposed = false;

    public bool IsDisposed { get { return _disposed; } }

    public AccountWriter(TextWriter target)
    {
        _namespaces = new XmlSerializerNamespaces();
        _namespaces.Add("", "");

        _ser = new XmlSerializer(typeof(Account));

        _writer = XmlWriter.Create(target);
    }

    public void Write(Account acct)
    {
        if (_disposed) throw new ObjectDisposedException("AccountWriter");

        if (!_wroteHeader)
        {
            _writer.WriteStartElement("BillingFile");
            _wroteHeader = true;
        }

        _ser.Serialize(_writer, acct, _namespaces);
    }

    public void Flush()
    {
        if (_disposed) throw new ObjectDisposedException("AccountWriter");
        _writer.Flush();
    }

    public void Dispose()
    {
        if (!_disposed)
        {
            if (_wroteHeader)
            {
                _writer.WriteEndElement();
                _wroteHeader = true;
            }

            _writer.Dispose();
            _disposed = true;
        }
    }
}

using (var writer = new AccountWriter(Console.Out))
{
    foreach (var acct in accounts)
    {
        writer.Write(acct);
    }
}