Question

I am saving a series of protobuf-net objects in a database cell as a Byte[] of length-prefixed protobuf-net objects:

//retrieve existing protobufs from database and convert to Byte[]
object q = sql_agent_cmd.ExecuteScalar();
older-pbfs = (Byte[])q;

// serialize the new pbf to add into MemoryStream m
//now write p and the new pbf-net Byte[] into a memory stream and retrieve the sum

var s = new System.IO.MemoryStream();
s.Write(older-pbfs, 0, older-pbfs.Length);
s.Write(m.GetBuffer(), 0, m.ToArray().Length); // append new bytes at the end of old
Byte[] sum-pbfs = s.ToArray();

//sum-pbfs = old pbfs + new pbf. Insert sum-pbfs into database

This works fine. My concern is what happens if there is slight db corruption. It will no longer be possible to know which byte is the length prefix and the entire cell contents would have to be discarded. Wouldn’t it be advisable to also use some kind of a end-of-pbf-object indicator (kind of like the \n or EOF indicators used in text files). This way even if a record gets corrupted, the other records would be recoverable.

If so, what is the recommended way to add end-of-record indicators at the end of each pbf.

Using protobuf-netv2 and C# on Visual Studio 2010.

Thanks
Manish

Answer 1

If you use a vanilla message via Serialize / Deserialize, then no: that isn’t part of the specification (because the format is designed to be appendable).

If, however, you use SerializeWithLengthPrefix, it will dump the length at the start of the message; it will then know in advance how much data is expected. You deserialize with DeserializeWithLengthPrefix, and it will complain loudly if it doesn’t have enough data. However! It will not complain if you have extra data, since this too is designed to be appendable.

In terms of Jon’s reply, the default usage of the *WithLengthPrefix method is in terms of the data stored exactly identical to what Jon suggests; it pretends there is a wrapper object and behaves accordingly. The differences are:

• no wrapper object actually exists
• the “withlengthprefix” methods explicitly stop after a single occurrence, rather than merging any later data into the same object (useful for, say, sending multiple discreet objects to a single file, or down a single socket)

The difference in the two “appendable”s here is that the first means “merge into a single object”, where-as the second means “I expect multiple records”.

Unrelated suggestion:

s.Write(m.GetBuffer(), 0, m.ToArray().Length);

should be:

s.Write(m.GetBuffer(), 0, (int)m.Length);

(no need to create an extra buffer)