Question

Trying to introduce caching/state-persisting into the middle tier of an application with a complex object layer (and a WCF service layer, not the focus here) running under IIS. Have settled on memcached/enyim as the caching architecture, and now need to get these objects serialized efficiently (speed and space) for it.

The object layer has a lot of pointers and interdependencies among objects, along these lines:

internal class SomeObj
{
    private string attr1;
    private int attr2;
    private OtherObj otherObj;
    private List<OtherOtherObj> otherObjs;
}

internal class OtherObj
{
    //...more attributes
}

internal class OtherOtherObj
{
    // you get the idea
}

Note that all fields are private. Also worth noting, most objects are internal, and many properties are either read-only (no set) or use set from a user perspective (i.e. make an object “dirty”), so cannot be used in rehydration. I have dozens of types that need caching.

I like the looks of both protobuf-net and msgpack, but I need to get the serialization done as fast as possible, with as little alteration of the existing architecture (which works well as is) as possible, and it seems like both of these have limited support for object hierarchies. I understand DTO-type serialization well, but am new to planning the right way to serialize an object for a cache. Can one of these tools work for me? Am I stuck using built-in .NET binary in order to work with a constructor, and repopulate attributes and objects on my own terms?

EDIT: just to clarify that last question–might make more sense if it read “Am I stuck using built-in .NET binary so that I can control the constructor, and repopulate attributes and objects on my own terms?

Answer 1

I can’t comment on msgpack, but yes: protobuf-net can do everything you’ve mentioned, including:

• serialize non-public types
• serialize private fields
• serialize trees
• constructor-skipping or user-provided factory (or just parameterless constructor)
• serialize full/cyclic graphs (by explicitly marking the affected as references)
• attribute usage, or runtime configuration without changing the DTO at all (although frankly, attributes are usually easier)
• fast and compact output
• inheritance support

In the case of the example given, the simplest “does it work” test would be to just make the types with [ProtoContract] (there’s an optional setting in that attribute for constructor-skipping), and mark the fields as [ProtoMember(n)], for example n=1,2,3,… (unique in each type, but does not need to be unique between types)

Other than the fact that we use Redis+BookSleeve rather than memcached+enyim, this is exactly what we do at Stack Exchange for object caching. Well, we also do a speculative gzip for large objects – if there is lots of string data, gzip can help shave off a few extra bytes.