Question

In platforms like Javascript, AS3 and Python which support and recommend untyped data arrays by default, arrays are usually the simplest and most effective way of storing arbitrary user data in memory. (tabular data, data from CSVs, data from JSON, etc)

.NET however likes everything in a strongly-typed format; you just can’t dump anything into a List<object> array, although it works, it will be slower and clumsier to handle (type checks, type casting every usage). So usually you end up defining the data schema as a class, with properties corresponding to the columns, and storing data in instances of that class.

So what are the recommended methods of storing arbitrary data in memory, especially when the schema keeps upgrading (such as CSV or JSON) that you cannot “hard-code” for at the time of development.

Edit: Such data may include numbers (int/float), strings, dates, times, units, geospatial data, geometric data, embedded files, essentially everything a MySQL database or JSON file could store.

Edit: While in memory, this data can and would be used for every kind of processing; calculations to generate charts, string processing to search for data by substrings, number crunching algorithms for geospatial/3D data, etc, optimization algorithms that validate dirty data and optimize redundant data, etc.

Answer 1

There are probably several ways to go here.

You could create classes programatically through reflection.

How to dynamically create a class in C#?

then use generic collections and some LINQ to Objects to query the data.

Or, you could use or create your own datasource or datatables on the fly from whatever datasource and make it the type you like depending on your logic, if its a database you could check the type of the column of a datatable. Then you could query the datatable with the select method of the datatable object.