Question

I have a powershell process that runs on demand which collects all “Request History” logs from an application via web service calls. The result requests are casted to objects and have NoteProperty values (attribute-value pair) end up in a large list array (usually 1400 items) per week.

What I wanted to do is store all of these requests for historical purposes so that the application doesn’t purge them on their own. Therefore I created a simple table on a database that stores all the attribute value pairs for each request that doesn’t already exist in my newly created database.

I then setup an OleDB connection in the powershell script to a MSSQL server and select all the records from the table to fill a DataTable (i’m not good with OleDB or DataTables). After-which I loop through each item in the list array to validate it doesn’t already exist in the DataTable. For each record that doesn’t exist I add a new row in the DataTable with the attribute-value pairs. From there I assume the command builder helps with the Insert statement so I don’t have to check each attribute-value if it’s null or blank or even write the query at all. Then finally I “Update” the OleDBAdapter with the newly appended DataTable.

While this process works, I realized that what it’s doing is pulling down all the data from the database and then comparing to my list array and re-committing the newly added records. The larger the database, the longer this takes. Is there anyway to quickly and more efficiently do this without having to write any SQL statements? I like how the CommandBuilder works for DataTables.

Below is the Function called to update the database after all the “Request History” objects have been fetched

function UpdateDatabase([Parameter(Mandatory=$true)] $allRequests)
{
    $objOleDbConnection = New-Object "System.Data.OleDb.OleDbConnection"
    $objOleDbCommand = New-Object "System.Data.OleDb.OleDbCommand"
    $objOleDbAdapter = New-Object "System.Data.OleDb.OleDbDataAdapter"
    $objDataTable = New-Object "System.Data.DataTable"

    $objOleDbConnection.ConnectionString = "Provider=SQLNCLI10;Server=SERVER;Database=DB1;Trusted_Connection=yes;"
    $objOleDbConnection.Open()

    $objOleDbCommand.Connection = $objOleDbConnection
    $objOleDbCommand.CommandText = "SELECT * FROM dbo.RequestLog"

    ##set the Adapter object and command builder
    $objOleDbAdapter.SelectCommand = $objOleDbCommand
    $objOleDbCommandBuilder = New-Object "System.Data.OleDb.OleDbCommandBuilder"
    $objOleDbCommandBuilder.DataAdapter = $objOleDbAdapter

    ##fill the objDataTable object with the results
    [void] $objOleDbAdapter.Fill($objDataTable)
    [void] $objOleDbAdapter.FillSchema($objDataTable,[System.Data.SchemaType]::Source)

    #store all the primary keys in a list for kicking out dups
    $sql_id = @()
    $objDataTable.Rows | foreach { $sql_id += $_.PKID}

    #####

    #loop through all the requests
    trap {
    "Error: $($i)"
    }
    $i = 0
    $total = $allRequests.count
    foreach ($request in $allRequests)
    {       
        $i++
        write-progress -activity "Filling DataTable" -status "% Complete: $($i/$total*100)" -PercentComplete ($i/$total*100)
        #check to see if entry already exists in our table (by primary key)
        if (!($sql_id -contains $request.PKID.Value))
        {
            #shouldn't have to do this but i noticed sometimes requests are duplicate in the list? (probably restarted the script and caught some old requests
            $sql_id += $request.PKID.Value

            $row = $objDataTable.Rows.Add($request.PKID.Value)
            #go through all the attributes from the request and add them to the table
            $list = get-member -in $request | Where-Object { $_.MemberType -eq "NoteProperty" }
            foreach ($attr in $list)
            {
                if ($request.($attr.name) -ne $null)
                {
                    $row.($attr.name) = $request.($attr.name)
                }
            }

        } else { 
            #PKID already in DB
        }

    }
    #update the database with our new records
    $objOleDbAdapter.Update($objDataTable)

    ## close the connection 
    $objOleDbConnection.Close()
}

Answer 1

You’re going to need to write a little T-SQL code to make the process more efficient. You’ll need to send the new rows to SQL Server so that processing takes place on SQL Server. One solution is to use Table Valued Parameters which allow you to pass a DataTable to SQL Server. I’ve blog an example here:

http://sev17.com/2012/04/appending-new-rows-only/