Question

I am looking for a way to pull XML from a SQL database and shred the XML via SSIS in bulk. I currently have a package that can pull XML from the database and pass the XML to a stored procedure, via variable, for shredding but this only works 1 record at a time. When processing 100,000 records, this can become quite time consuming.

I would like to shred multiple XML values at once using SSIS. Is this possible with SSIS? Perhaps something in a Data Flow Task where all the XML values are selected from a source then passed to a parser of some sort?

Answer 1

So after a lot of searching and thinking about the problem, I redesigned the stored procedures that did the shredding. Instead of shredding a variable being passed into the stored procedure, I just shredded from the XML column itself. This allowed me to shred many XML values at once instead of looping and passing them to a stored procedure one at a time. This gave me the performance boost I was looking for. So instead of something like this…

SET @xmlData = CAST(@xmlMessageData AS XML)

SELECT
   , row.value('xmlNode1[1]', 'BIT' )                   AS Column1  
   , row.value('xmlNode2[1]', 'NVARCHAR(255)')          AS Column2
   , row.value('xmlNode3[1]', 'BIT' )                   AS Column3
   , row.value('xmlNode4[1]/Name[1]', 'NVARCHAR(255)' ) AS Column4
FROM @xmlData.nodes('xmlRootNode') AS T1(row)

I now do this…

SELECT
   , row.value('xmlNode1[1]', 'BIT' )                   AS Column1  
   , row.value('xmlNode2[1]', 'NVARCHAR(255)')          AS Column2
   , row.value('xmlNode3[1]', 'BIT' )                   AS Column3
   , row.value('xmlNode4[1]/Name[1]', 'NVARCHAR(255)' ) AS Column4
FROM [ESBMessagesData] D
CROSS APPLY
    [DataItem].nodes('xmlRootNode') AS T1(row)