Question

Part of my current project involves populating an Excel worksheet from a large SQL table, which contains 120,000+ records. I’m currently using SqlDataReader with a while loop to directly populate a new Excel sheet, but obviously this takes an unrealistic amount of time. Building a local .CSV file with StringBuilder then opening that in Excel seems to be a faster option. Is it preferred to dump the data to file every thousand rows or so, or write it all at once? Or is there a more efficient way to import this data that I’m not considering?

The dataset is different depending on user input, so I can’t simply append to an existing file.

Current code:

SqlConnection cnnct = new SqlConnection("Data Source=Source;Initial Catalog=Catalog;Integrated Security=True;");
            cnnct.Open();
            SqlCommand cmd = new SqlCommand();
            cmd.Connection = cnnct;
            string cmdText = "SELECT * FROM TABLE";
            cmd.CommandText = cmdText;

            SqlDataReader dr = cmd.ExecuteReader();
            while (dr.Read())
            {
                columnChar = 'A';
                rowCount++;

                cellName = columnChar.ToString() + rowCount.ToString();
                    ws.get_Range(cellName, cellName).Value2 = dr["EXAMPLE_FIELD"].ToString();
                    columnChar++;
                //The above three lines are identical for each field (there are 26)
            }

Answer 1

Well, I would avoid Excel interop at all costs. Your idea of just writing a .CSV file has lots of merit. As always with “best” performance it depends on environment, but unit tests always help.

System.IO is pretty optimal. I would just read a row from SqlDataReader and write the CSV line. It becomes very simple code and I bet is fairly “fast”.

Good luck.