Question

I am looking for a way to add a serialized value of an object (byte[]) to a repeated field in a protocol buffers message.

I have an application that keeps data items in serialized form in memcached and needs to deliver them to remote clients. The client requests data items by providing a list of keys and the server sends back the list of data items. The content of data items is not important for the server; it does not need to know what is contained in them, it only needs to know their key.

The current approach is to fetch items from memcached, deserialize them, add them to the list of data items in a response, serialize the response into a byte array and send it over a socket. This is not optimal, because we deserialize data items only to have them serialized again in the next step. Both serializations (for memcached and for output) are done with protocol buffers. Ideally, we could skip the deserialization after fetching the data from memcached and add the serialized values to the response. I looked into both protobuf-net and protobuf-csharp and did not find a way to accomplish this. Is it possible? Did I overlook something?

Here are proto definitions (simplified):

message Request {    
    required int32 messageId;
    repeated string keys;
}

message DataItem {

    required string key = 1;
    required ValueType type = 2;      // the type of the value, enumeration

    optional int32 intValue = 3;
    optional int64 longValue = 4;
    optional double doubleValue = 5;
    optional float floatValue = 6;
    optional bool boolValue = 7;
    optional string stringValue = 8;
}

message Response {
    required int32 messageId;
    repeated DataItem dataItems;
}

Answer 1

Well, the bytes field type represents opaque binary data… is that what you’re looking for? Note that for immutability purposes (in protobuf-csharp anyway) these are represented as immutable ByteString values – but you’ll be able to copy those from one message to another without the actual data being copied (i.e. keep a single reference to the same blob in two messages).