Question

Background

I am receiving CSV data files from clients that contain a large amount of data that I don’t need and a small amount of data that I do. In the future I may need to access that data and although I’m archiving the original data files I was hoping for something a bit easier to query. I was hoping for a solution that didn’t mean the data files stayed in the same format – i.e. the client may add/remove columns and I don’t want my implementation to bail on missing data or fail to archive additional data.

As I’m building the application in Azure, Azure table storage looked correct to me – I could read the data files and then store whatever key/value pairs I read into the data store.

Upshot

I would like to know how to store a Dictionary<K, V> or Hashtable or some other key/value pairs in Azure.

Answer 1

By overriding the ReadEntity and WriteEntity methods on classes derived from TableEntity, additional properties can be stored.

Here’s my naive implementation

public class TestEntity : TableEntity {

    public TestEntity(string a, string b) {
        PartitionKey = a;
        RowKey = b;
        DataItems = new Dictionary<string, string>();
    }

    public Dictionary<string, string> DataItems { get; set; }

    public override IDictionary<string, EntityProperty> WriteEntity(OperationContext operationContext) {
        var results = base.WriteEntity(operationContext);
        foreach (var item in DataItems) {
            results.Add("D_" + item.Key, new EntityProperty(item.Value));
        }
        return results;
    }

    public override void ReadEntity(IDictionary<string, EntityProperty> properties, OperationContext operationContext) {
        base.ReadEntity(properties, operationContext);

        DataItems = new Dictionary<string, string>();

        foreach (var item in properties) {
            if (item.Key.StartsWith("D_")) {
                string realKey = item.Key.Substring(2);
                ItemData[realKey] = item.Value.StringValue;
            }
        }
    }
}

I’ve noted that Azure table storage can only store a total of 255 key/value pairs – or 252 custom ones once PartitionKey, etc. is taken into account, so this has to be handled somewhere as well.