Question

I am trying to create a simple in-memory grid that can be serialized using protocol buffers.

The idea is the user can create/define columns of any type (primitives or user defined as long as they’re protocol buffers tagged).

My problem is you can’t serialize Type data with protocol buffers so how can i achieve this?

Code is below showing how i’d hoped to code the Columns of the grid.

Thanks a lot.

[TestClass]
public class UnitTest1
{
    [TestMethod]
    public void TestMethod1()
    {
        Column colOrderId = new Column("OrderId", typeof(uint));
        Column colOrderDesc = new Column("OrderDesc", typeof(string));
        Column colPrice = new Column("Price", typeof(Price));
        Column colOrderDateTime = new Column("OrderDateTime", typeof(DateTime));

        var s = colOrderId.ToBArray();
    }
}

[ProtoContract, Serializable]
public sealed class Column
{
    public Column(string name, Type type)
    {
        Name = name;
        Type = type;
    }

    [ProtoMember(1)]
    public string Name { get; private set; }

    [ProtoMember(2)]
    public Type Type { get; private set; }
}

[ProtoContract, Serializable]
public sealed class Price
{
    public Price(double value, string currency)
    {
        Value = value;
        Currency = currency;
    }

    [ProtoMember(1)]
    public double Value { get; private set; }

    [ProtoMember(2)]
    public string Currency { get; private set; }
}

public static class ProtoBufEx
{
    public static byte[] ToBArray<T>(this T o)
    {
        using (MemoryStream ms = new MemoryStream())
        {
            ProtoBuf.Serializer.Serialize(ms, o);
            return ms.ToArray();
        }
    }
}

Answer 1

There’s a couple of issues in this approach; yes, in standard protocol buffers any kind of Type metadata is a bit naughty, as it isn’t really interoperable, but I play a little loose with that in v2 on an opt-in basis – allowing you to send object etc as long as internally it still knows what you mean.

This, however, will get expensive for cell-by-cell work, even with the optimisations built in (it only sends the same type meta once, for example).

IMO, though, a far better option is to restrict yourself to a list of known types that are all known ahead of time. If you can do that there are some interesting tricks with inheritance that are pretty efficient and reasonably suited for this scenario. But essentially you’d have something like:

[ProtoContract]
[ProtoInclude(4, typeof(Column<int>))] // etc
abstract class Column {
    [ProtoMember(1)]
    public string Name {get;private set;}

    public abstract object Value {get;private set;}
}
[ProtoContract]
class Column<T> : Column {
    [ProtoMember(1)]
    public T TypedValue { get;private set;}
    override Value {...shim to TypedValue...}
}
// etc

(pseudo-code, incomplete)

I’d happily walk you through a lot of that, but you might also want to look at this blog entry to see if using DataTable (much as I dislike it for data-access) might save some effort here.