Question

I have a very large dataset which I am currently writing out to a text file (IO). It is very slow and causing the system to chew up a lot of resources as there are 10’s of thousands of rows.

I’m wondering if anybody can recommend a good way to do this to reduce the load on my system or at least smooth out the process to avoid big spikes in demand for memory resources etc. I don’t mind if it means it takes longer, but as long as it’s not putting too much load on the machine.

Answer 1

Your question hardly makes sense, but assuming you are reading the results from database in chunks you could write them in chunks to the file to avoid loading the entire dataset in memory, just like so:

using (var conn = new SqlConnection(SomeConnectionString))
using (var cmd = conn.CreateCommand())
{
    conn.Open();
    cmd.CommandText = "SELECT foo, bar FROM baz;";
    using (var reader = cmd.ExecuteReader())
    {
        using (var writer = new StreamWriter("result.txt"))
        {
            while (reader.Read())
            {
                var foo = reader.GetString(reader.GetOrdinal("foo"));
                var bar = reader.GetInt32(reader.GetOrdinal("bar"));
                writer.WriteLine(string.Format("{0}, {1}", foo, bar));
            }
        }
    }
}

In terms of memory consumption this will Rock’N’Roll and in terms of performance it would of course depend on the optimization of your SQL query and the capabilities of your SQL server.