Question

I have a large amount of data stored in an XML file, 173 MB (4.6 million lines), that I have stored in my Windows Forms application’s working directory. It is the result of writing a datatable to an XML file. The datatable was originaly populated from a query to a SQL server.

The reason that I have it stored locally rather than requesting it from the server is that the data request took upwards of 40 seconds and at times timed out and the data is static and will never change, moreover the user can be offline and still use the data.

Loading the file back into the data table takes 20-30 seconds. I am not too woried about the time that it took to load from disk as I let the user know that data is loading and to be patient. However I don’t like the XML file format and I am looking for other ideas for disk storage.

The data table is only beng used as a middleman for the eventual population of a collection object. If you have sugestions I would like to hear them.

I am hoping to stay away from a database solution and lean towards a binary file approach. Below is my first attempt, but I get an out of memory exception:

byte[] b = null;

using (MemoryStream stream = new MemoryStream())
{
   BinaryFormatter bformatter = new BinaryFormatter();
   bformatter.Serialize(stream, timeData);
   b = stream.ToArray();
}

using (FileStream fileStream = new
   FileStream("brad.bin", FileMode.Create, FileAccess.Write))
{
   fileStream.Write(b, 0, b.Length);
}

Answer 1

I’d look at a compact (local) database such as SQL Server CE or SQLite. Databases are designed for exactly this.