Question

I’m calling a third party web service that returns a large amount of XML (1000s of rows). What is the best way to get the XML into an underlying SQL server 2008 R2 table. At the moment I am retrieving the XML and deserializing into .NET class collections using the associated XSD. I then iterate through the collection making multiple calls to a INSERT stored proc. The performance isn’t too bad but I’m sure there must be a quicker way to do this. Should I be looking at SqlBulkCopy? I’ve heard that LINQ-TO-SQL is slow when it comes to bulk inserts/updates. Any advice would be welcome.

Answer 1

Read about SQLXMLBulkLoad4, that seems to handle bulk inserts of XML to sql server. I havent tried it myselfe, but its worth a try.

If that not working, then check out System.Data.SqlClient.SqlBulkCopy
Using SqlBulkCopy I put 200.000 rows of data read from a file in about 4 seconds.

The following code takes a filename as argument, read all lines and transform the lines to a datarow that is added to a datatable (The datatable has to have same format as the target table, Im using a strongly typed datatable in this case). When the datatable contains 1000 rows, I write it to the server with the WriteToServer method. Then empty the datatable and start over.

you could easy change it to transform rows from your xml-document instead.

Imports System.IO
Imports System.Data.SqlClient

    Public Sub ParseFile(ByVal filename As String)
    Using BulkCopy As SqlBulkCopy = New SqlBulkCopy(My.Settings.DBConnectionString, SqlBulkCopyOptions.TableLock)
        BulkCopy.DestinationTableName = "Destinationtable"
        Dim TableDataSet As New MyDS.UserDataTable
        Try
            Using sr As StreamReader = System.IO.File.OpenText(filename)
                Dim l As String = Nothing
                l = sr.ReadLine()
                If Not l Is Nothing Then AddToDS(l, TableDataSet)
                Do Until l Is Nothing
                    l = sr.ReadLine()
                    If Not l Is Nothing Then AddToDS(l, TableDataSet)
                    If TableDataSet.Rows.Count > My.Settings.BulkSize Then 'bulk copy every 1000 rows to db.
                        BulkCopy.WriteToServer(TableDataSet)
                        TableDataSet.Rows.Clear()
                    End If
                Loop
                If TableDataSet.Rows.Count > 0 Then 'write the rest
                    BulkCopy.WriteToServer(TableDataSet)
                    TableDataSet.Rows.Clear()
                End If
                sr.Close()
            End Using
        Catch ex As Exception
            MsgBox("Error:" & ex.Message)
        End Try
    End Using
End Sub

Sub AddToDS(ByVal s As String, ByRef ds As MyDS.UserDataTable)
    'row should contain username + tab + carname
    Dim arr() As String = s.Split(vbTab)
    If arr.Count < 2 Then Exit Sub
    ds.AddUserRow(arr(0).Trim, arr(1).Trim)
End Sub