Question

I was troubleshooting an application today at work that runs periodically to handle some data processing needs, it often runs just fine but sometimes the application will crash and someone will have to manually restart it.

After looking at the source and some debugging it seems the core of the program relies on a recursive function that is removing rows from a DataTable and then stopping once the DataTable no longer has any rows remaining.

Here is the code:

Public Sub ProcessData(ByRef dtTable As DataTable)
  Dim DataView1 As New DataView()
  Dim UID As String = ""
  Dim dtUID As New DataTable

  dtUID = dtTable.Clone()
  UID = dtTable.Rows(0)("UID")    

  DataView1 = dtTable.DefaultView

  Dim expression As String = "UID ='" & UID & "'"
  DataView1.RowFilter = expression

  For n As Integer = 0 To DataView1.Count - 1
      dtUID.ImportRow(DataView1.Item(n).Row)
  Next

  Dim foundRows() As DataRow
  foundRows = dtTable.Select(expression)

  For n As Integer = 0 To foundRows.GetUpperBound(0)
      dtTable.Rows.Remove(foundRows(n))
  Next

  For n As Integer = 0 To dtUID.Rows.Count - 1
      ProcessRecord(dtUID, n)
  Next

  If dtTable.Rows.Count > 0 Then
      ProcessData(dtTable)
  End If
End Sub

ProcessRecord() is a function that applies all the business logic to the record and also does a write to a database. The unhandled stack overFlow exception happens at:

For n As Integer = 0 To DataView1.Count - 1

In the real world this function would be fed a DataTable with around 100,000 records in it. In testing it, it will run through about 40,000-65,000 pretty consistently after many trial runs before it has the stack overflow. The code that creates the DataTable and ProcessRecord() flag records that have already been processed, so if you’re working through a 100,000 record large data set then subsequent calls to this method after a crash will be passed a smaller DataTable (35,000-60,000 records.) Interestingly the stack overflow will happen more and more often with smaller DataTables, I can consistently get through 40k-65k of a 100k DataTable but cannot consistently get through 15k of a 15k DataTable without a crash (eventually it will complete all records, with multiple crashes over the last few thousand.)

Once I saw what the problem was I basically scrapped most of this code and used an implementation I’ve used for similar tasks in the past. But I really wanted to understand exactly why this code was causing this problem.

Answer 1

The problem is simply that it is a recursive function.

Each function invocation uses stack space to store its parameters and local variables, and stack space is limited.

There’s no reason that this particular operation should be recursive. In fact, when the maximum number of recursions cannot be estimated or limited easily, it’s a bad idea to use a recursive solution.