Question

I am writing a C# application that needs to read about 130,000 (String, Int32) pairs at startup to a Dictionary. The pairs are stored in a .txt file, and are thus easily modifiable by anyone, which is something dangerous in the context. I would like to ask if there is a way to save this dictionary so that the information can be reasonably safely stored, without losing performance at startup. I have tried using BinaryFormatter, but the problem is that while the original program takes between 125ms and 250ms at startup to read the information from the txt and build the dictionary, deserializing the resulting binary files takes up to 2s, which is not too much by itself but when compared to the original performance is a 8-16x decrease in speed.

Note: Encryption is important, but the most important should be a way to save and read the dictionary from the disk – possibly from a binary file – without having to use Convert.ToInt32 on each line, thus improving performance.

Answer 1

interesting question. I did some quick tests and you are right – BinaryFormatter is surprisingly slow:

• Serialize 130,000 dictionary entries: 547ms
• Deserialize 130,000 dictionary entries: 1046ms

When I coded it with a StreamReader/StreamWriter with comma separated values I got:

• Serialize 130,000 dictionary entries: 121ms
• Deserialize 130,000 dictionary entries: 111ms

But then I tried just using a BinaryWriter/BinaryReader:

• Serialize 130,000 dictionary entries: 22ms
• Deserialize 130,000 dictionary entries: 36ms

The code for that looks like this:

public void Serialize(Dictionary<string, int> dictionary, Stream stream)
{
    BinaryWriter writer = new BinaryWriter(stream);
    writer.Write(dictionary.Count);
    foreach (var kvp in dictionary)
    {
        writer.Write(kvp.Key);
        writer.Write(kvp.Value);
    }
    writer.Flush();
}

public Dictionary<string, int> Deserialize(Stream stream)
{
    BinaryReader reader = new BinaryReader(stream);
    int count = reader.ReadInt32();
    var dictionary = new Dictionary<string,int>(count);
    for (int n = 0; n < count; n++)
    {
        var key = reader.ReadString();
        var value = reader.ReadInt32();
        dictionary.Add(key, value);
    }
    return dictionary;                
}

As others have said though, if you are concerned about users tampering with the file, encryption, rather than binary formatting is the way forward.