Question

I am using the following class to serialize individual objects (T) with a prefix, and then attempting to deserialize back to a list w/ the DeserializeItems() method:

public class ProtobufSerializationProvider<T> : ISerializationProvider<T>
{
    readonly Type type; 
    readonly TypeModel model;

    public ProtobufSerializationProvider()
    {
        this.type = typeof(T); 
        this.model = createModel(this.type);
    }

    public byte[] Serialize(T instance)
    {
        byte[] buffer; 
        using (MemoryStream strm = new MemoryStream())
        {
            model.SerializeWithLengthPrefix
              (strm, instance, type, PrefixStyle.Base128, 1);

            buffer = strm.GetBuffer();              
        }
        return buffer; 
    }

    // here is the problem method
    public IEnumerable<T> DeserializeAll(MemoryStream stream)
    {
        return model.DeserializeItems<T>(stream, PrefixStyle.Base128, 1); 
    }

    TypeModel createModel(Type type)
    {
        try
        {
            RuntimeTypeModel runtimeModel = TypeModel.Create();
            this.addTypeToModel(runtimeModel, type);
            this.addTypePropertiesToModel(runtimeModel, type);
            return runtimeModel.Compile();
        }
        catch (Exception e)
        {
            throw e.InnerException;
        }
    }

    void addTypePropertiesToModel(RuntimeTypeModel typeModel, Type type)
    {
        PropertyInfo[] properties = type.GetProperties();
        for (int i = 0; i < properties.Length; i++)
        {
            Type innerType = properties[i].PropertyType;
            if (!innerType.IsPrimitive && !(innerType == typeof(string)))
            {
                addTypeToModel(typeModel, properties[i].PropertyType);
            }
        }
    }

    MetaType addTypeToModel(RuntimeTypeModel typeModel, Type t)
    {                
        var properties = t.GetProperties()
            .Select(p => p.Name)
            .OrderBy(name => name);

        return typeModel
            .Add(t, true)
            .Add(properties.ToArray());            
    }
}

I am getting a default InvalidOperationException “Operation is not valid due to the current state of the object” when I try to enumerate the IEnumerable, whether by casting ToList(), or counting it Count(), etc. Specifically the MoveNext() method will throw the error:

enumerator.MoveNext()

Also the stream has to be open until DeserializeItems(stream) returns, and I ensured that was the case. But once the IEnumerable successfully returns I cannot use it.

Not sure if there is a problem with the serialized items or not. Also I noticed that each of my items 256 bytes, although a large portion of those bytes are just trailing nulls.

This is the class I am serializing as a test. Note that I am not using attributes since I create the model manually:

public class NestedFoo
{
    public string NestedFooStr { get; set; } 
}

public class Foo
{
    public string Foo1 { get; set; }
    public string Foo2 { get; set; }
    public string Foo3 { get; set; }
    public string Foo4 { get; set; }

    public NestedFoo NestedFoo { get; set; }
}

Thanks.

UPDATE

After switching from MemoryStream.GetBuffer() to MemoryStream.ToArray(), the message is shortened but trying to cast the IEnumerable .ToList() or any other operation produces the same error.

However, I have noticed that after switching to .ToArray() the ‘Current’ property of the IEnumerable after enumeration reaches the last element in the IEnumerable before throwing the error, whereas when using .GetBuffer() the ‘Current’ property on the first element when the error is thrown.

I suspect the problem could be that the IEnumerable<> doesn’t know when its out of elements because its reached the end of the stream? I append several items serialized with SerializeWithLengthPrefix() to a single byte[], and then there may be nulls left over trailing the array. If I read that entire array into the DeserializeItems method, does it need to know how to terminate or does it only continue if it detects a length prefix (as I would expect?).

Also one other question I had was to what extent using the SerializeWithLengthPrefix() method is equivalent to just serializing a List in one operation, using the DataFormat.Group enumeration, like:

[ProtoContract]
public class ProtoList<T>
{
    [ProtoMember(1, DataFormat = DataFormat.Group)]
    public List<T> List { get; set; }

    public ProtoList() { }

    public ProtoList(IEnumerable<T> items)
    {
        this.List = new List<T>(items); 
    }
}

Using a container such as this seems to perform very well, so I may just go with this route in cases where I want to serialize a list in one shot.

However for serializing items individually as they are added I am currently using my own prefixing function, so if I can get the Protobuf prefixing methods to work that would be good too.

Answer 1

The 256/trailing nulls is because you are using MemoryStream.GetBuffer(). This method is fine as long as it is used in combination with the Length, as the buffer is oversized. If you want a right-sized array, use ToArray() instead. Alternatively use ArraySegment<byte> to avoid copying any data.

It is possible this also fixes the exception (0 is not a valid value for a field-header). If not, leave a comment so I know to look in more detail. If it does fix it, please also let me know, and I’ll try to make the error message (for the 0 field-header case) more helpful.