Question

I have a DataTable with multiple columns including AccountNumber, Year, and Month. I am exporting the information in this table to an Excel workbook. Since this table has the potential to be extremely large, I must check the number of records in the table (since Excel can only have 65536 rows or something like that. If the original table is small enough, I need to put all of the records in a single worksheet. If there are too many records for one worksheet, I need to separate the records into multiple worksheets based on AccountNumber. Additionally, if there are too many records for a specific AccountNumber then I need to split that AccountNumber into multiple worksheets based on Year. If a specific Year still has too many records then I have to split them by Month.

For example:

If I have a total of 500,000 records, I must split them by AccountNumber getting:

Worksheet Name —- Number of Records

• 1111 —- 150,000
• 2222 —- 50,000
• 3333 —- 100,000

Then I would need to split Accounts 1111, and 3333 into multiple worksheets based on the Year. I would then have something like this:

Worksheet Name —- Number of Records

• 1111 – 2010 —- 50,000
• 1111 – 2011 —- 100,000
• 2222 —- 50,000
• 3333-2010 —- 50,000
• 3333-2011 —- 50,000

Then, Since 1111 – 2011 is still too large I would have to split that one based on Month, finally giving:

Worksheet Name —- Number of Records

• 1111-2010 —- 50,000
• 1111-201105 —- 30,000
• 1111-201106 —- 30,000
• 1111-201107 —- 40,000
• 2222 —- 50,000
• 3333-2010 —- 50,000
• 3333-2011 —- 50,000

The code that is being used to create the Excel file is a project that was written by my company. To make it simple, the function accepts a DataTable and writes out the records in the DataTable in the format of an Excel spreadsheet. Any ideas on how I can do this without making anymore calls to the database? Thanks in advance.

Answer 1

You can use LINQ-to-DataSet to group the DataRows accordingly and add them to separate DataTables with less rows than 65536.

Have a look at this working sample which also generates the correct File/DataTable-Names:

Private Shared sampleAccountNumbers As List(Of Int32) = {1111, 2222, 3333, 4444}.ToList
Private Const MAX_ROWS = 65536
Private tblSource As New DataTable()
Private allResultTables As New List(Of DataTable)

Private Sub SplitTableRows()
    Dim accQuery = _
        From row In tblSource
        Group row By AccountNumber = row("AccountNumber") Into AccNumGroup = Group
    For Each acc In accQuery
        If acc.AccNumGroup.Count > MAX_ROWS Then
            Dim yearQuery = _
               From row In acc.AccNumGroup
               Group row By Year = row("Year") Into YearGroup = Group
            For Each y In yearQuery
                If y.YearGroup.Count > MAX_ROWS Then
                    Dim monthQuery = _
                       From row In y.YearGroup
                       Group row By Month = row("Month") Into MonthGroup = Group
                    For Each m In monthQuery
                        If m.MonthGroup.Count > MAX_ROWS Then
                            'split by days or whatever...'
                        Else
                            Dim tblMonth = m.MonthGroup.CopyToDataTable()
                            tblMonth.TableName = String.Format("{0}-{1}{2}", acc.AccountNumber, y.Year, m.Month)
                            allResultTables.Add(tblMonth)
                        End If
                    Next
                Else
                    Dim tblYear = y.YearGroup.CopyToDataTable()
                    tblYear.TableName = String.Format("{0}-{1}", acc.AccountNumber, y.Year)
                    allResultTables.Add(tblYear)
                End If
            Next
        Else
            Dim tblAcc = acc.AccNumGroup.CopyToDataTable()
            tblAcc.TableName = acc.AccountNumber
            allResultTables.Add(tblAcc)
        End If
    Next
End Sub

Private Sub BtnStart_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles BtnStart.Click
    InitSourceDataTable()
    If tblSource.Rows.Count > MAX_ROWS Then
        SplitTableRows()
    Else
        allResultTables.Add(tblSource)
    End If
    For Each tbl In allResultTables
        'call your generate-excel-method...'
    Next
End Sub

Private Sub InitSourceDataTable()
    Dim rndAccNum As New Random(Date.Now.Millisecond)
    Dim rndYear As New Random(Date.Now.Millisecond)
    Dim rndMonth As New Random(Date.Now.Millisecond)
    tblSource.Columns.Add(New DataColumn("ID", GetType(Integer)))
    tblSource.Columns.Add(New DataColumn("AccountNumber", GetType(Integer)))
    tblSource.Columns.Add(New DataColumn("Year", GetType(Integer)))
    tblSource.Columns.Add(New DataColumn("Month", GetType(Integer)))
    For i As Int32 = 1 To 500000
        Dim newRow = tblSource.NewRow
        newRow("ID") = i
        newRow("AccountNumber") = sampleAccountNumbers(rndAccNum.Next(0, sampleAccountNumbers.Count))
        newRow("Year") = rndAccNum.Next(2005, 2012)
        newRow("Month") = rndMonth.Next(1, 13)
        tblSource.Rows.Add(newRow)
    Next
End Sub

This sample uses randomly 4 static AccountNumbers in the period 2005-2012 with 500.000 records. It generates 28 DataTables and clones all DataRows in only 1,6 seconds since LINQ-to-DataSet is working entirely in memory.