Question

I am using Protobuf-net (v1 r282) in a .NET CF 3.5 application(client) and a .NET 2.0 WebService.

There is a lot of data that needs to be sent between the client and server (for which protobuf-net is really great in reducing the time/amount of data being sent over the wire – reduces 5-7 mins of transfer time to just 25-30 seconds).

I am getting the MissingMethodException after around 13-15 calls to the Serializer.Deserialize method in the hand held device (CF 3.5):

StackTrace:

   at ProtoBuf.Property.PropertyFactory.Create[T](MemberInfo member)
   at ProtoBuf.Serializer`1.Build()
   at ProtoBuf.Serializer`1.DeserializeChecked[TCreation](SimpleWrapper& instance, SerializationContext source)
   at ProtoBuf.SerializerSimpleProxy`1.Deserialize(List`1& value, SerializationContext source)
   at ProtoBuf.Serializer.Deserialize[T](SerializationContext source)
   at ProtoBuf.Serializer.Deserialize[T](Stream source)
   at Lazurde.LazurdeDialogs.DLG_9600.DeserializeProtobufData[T](Byte[] protoArray)

If I try using the Serializer.DeserializeWithLengthPrefix method, after approx. the same number of calls the device just hangs.
Changing the order of the calls (bigger data deserialization first or at the end) does not make a difference.

I am using a MemoryStream for the Serialization/Deserialization process.

    public List<T> DeserializeProtobufData<T>(byte[] protoArray)
    {
        List<T> list = null;
        using (var stream = new MemoryStream(protoArray))
        {
            list = Serializer.DeserializeWithLengthPrefix<List<T>>(stream, PrefixStyle.Base128);
            //list = Serializer.Deserialize<List<T>>(stream);
            stream.Close();
        }
        return list ?? new List<T>();
    }

The byte[] protoArray specified in the stream for the largest one is approx. 900kb
The data is a List<T> where T are objects containing simple value types (and string and DateTime). The objects are specified as concrete types (not interfaces) and do not have any inheritance hierarchy.

If Serializer.Serialize method (approx. 10 calls) is used before deserialization,
I also sometimes get OutOfMemory Exception when using the Serializer.Deserialize after around 3-4 calls if using big data arrays first or 7-8 calls if later (but not necessarily on the big data deserialization even before that) .

This makes me think that there is some kind of memory leak in my app or it is using too much mem or there is too much fragmentation in the Large Object Heap when deserializing using protobuf but what I cannot understand is that it does not crash or throw exceptions when NOT using protobuf-net. (that is waiting around 10-15 mins till the data is transferred which using protobuf is less than a min.)

I also tried using GC statemnts to see if it helps after every deserialization but it does not seem to have any effect which makes me think too much fragmentation with Large Object Heap.

GC.Collect();
GC.WaitForPendingFinalizers();
GC.Collect();

I am profiling the memory and trying to find problem areas and any help will be much appreciated.
A recommendation of a good mem profiler for CF 3.5 will also be helpful (currently using .NETCF CLR Profiler)

The data is in a lot of different collections(approx. 15 for sending client to server and 20 for sending server to client)
The size of 3 of the (Server-to-Client) collections is approx 10000 records(could increase to 30000 in production) with each record having upto 45 fields.

Sorry for making this such a big post but I thought it is better to provide more info. rather than less.

Also, is there v2 for protobuf-net available for CF3.5? It’s not there in the r480 zip.

Help will be much appreciated.

Answer 1

Yes, MissingMethodException on CF is a royal PITA, and is actually one of the various reasons I wrote the entire core of protobuf-net, essentially from scratch.

I believe this should be fixed in v2, since it doesn’t make any generic/non-generic flips. If I didn’t build for CF, then that was simply an oversight (a bit of a pain, but I need to use different VMs to build the CF versions). It should build from source, else I can try and do a CF build when I get a moment.

Re any fragmentation: I’d rather not speculate on v1, since my advice would be “switch to v2”, and v2 has a very different code-base.