Question

I’m trying to serialize a large collection of objects (20,000) objects within the collection. I’m doing this using the following code:

XmlSerializer xs = new XmlSerializer(deserialized.GetType());
StringWriter sw;
using (sw = new StringWriter())
{
   xs.Serialize(sw, deserialized);   // OutOfMemoryException here
}

string packet = sw.ToString();
return packet;

Is there a better way of doing this, or am I doing something blatantly wrong?

Answer 1

It looks like it should work, but CF does have unpredictable limitations.

Is xml a requirement? I can’t remember trying it with 20k records, but another option might be to try using a different serializer – for example, protobuf-net works on CF2. I can’t guarantee it’ll work, but it might be worth a shot.

(in particular, I’m currently refactoring the code to try to work around some additional “generics” limitations within CF – but unless you have a very complex object model this shouldn’t affect you).

---

Example showing usage; note that this example also works OK for XmlSerializer, but protobuf-net uses only 20% of the space (or 10% of the space if you consider that characters are two bytes each in memory):

using System;
using System.Collections.Generic;
using System.IO;
using System.Xml.Serialization;
using ProtoBuf;

[Serializable, ProtoContract]
public class Department
{
    [ProtoMember(1)]
    public string Name { get; set; }
    [ProtoMember(2)]
    public List<Person> People { get; set; }
}

[Serializable, ProtoContract]
public class Person
{
    [ProtoMember(1)]
    public int Id { get; set; }
    [ProtoMember(2)]
    public string Name { get; set; }
    [ProtoMember(3)]
    public DateTime DateOfBirth { get; set; }
}


static class Program
{
    [MTAThread]
    static void Main()
    {
        Department dept = new Department { Name = "foo"};
        dept.People = new List<Person>();
        Random rand = new Random(123456);
        for (int i = 0; i < 20000; i++)
        {
            Person person = new Person();
            person.Id = rand.Next(50000);
            person.DateOfBirth = DateTime.Today.AddDays(-rand.Next(2000));
            person.Name = "fixed name";
            dept.People.Add(person);
        }

        byte[] raw;
        using (MemoryStream ms = new MemoryStream())
        {
            Serializer.Serialize(ms, dept);
            raw = ms.ToArray(); // 473,399 bytes
        }

        XmlSerializer ser = new XmlSerializer(typeof(Department));
        StringWriter sw = new StringWriter();
        ser.Serialize(sw, dept);
        string s = sw.ToString(); // 2,115,693 characters
    }
}

Let me know if you want more help – I can talk about this subject all day ;-p
Note that it can work just from the standard xml attributes ([XmlElement(Order=1)]) – I’ve used the more specific [ProtoMember(1)] etc for clarity. This also allows fine-grained control of serialization (zigzag vs twoscompliment, grouped vs length-prefixed, etc).