Question

I am writing some Natural Language Processing code and want to have a binary format dictionary to lookup word meanings. I originally used the standard Serialization and Binary Formatter mechanism but was horrified at how big the resulting file was, and how full of (redundant) type information. I have been trying to get the protobuf-net library to work for me but have the following problem:

When I write my dictionary to file I also put the offset that a record starts into an index file and I think that the serialization works OK, but when I look an entry up I get the correct offset but the DictionaryEntry’s Definitions property contains all of the DictionaryDefinitions from then on (thousands of them). The first DictionaryDefinition in the List is the correct one for the term searched for.

I have tried implementing this as both a Property and as a Public member, to no success. I have also tried SkipConstructor = true in the ProtoContract attribute and OverwriteList = true in the ProtoMember attribute.

[ProtoContract] 
public class DictionaryEntry
{
    [ProtoMember(1)]
    public string EntryKey { get; set; }
    [ProtoMember(2)]
    public List<DictionaryDefinition> Definitions{ get; set; }
}

[ProtoContract]
public class DictionaryDefinition
{
    [ProtoMember(1)]
    public string PartOfSpeech { get; set; }
    [ProtoMember(2)]
    public string Definition { get; set; }
}

I don’t think that I’ve done anything particularly dumb in the rest of my code, but if anyone could suggest anything for this part, I’d like to exclude this.

Thanks,

M

Answer 1

The raw protobuf format has neither a length prefix not a terminator – it is designed such that append === merge.

If you need to identify individual objects inside a single stream, use the *WithLengthPrefix methods for serialize/deserialize. This will allow you to reliably pick individual elements from the stream.