Question

I have got a table containing rows of related data which I need to transform within an SSIS package and I am not sure how to go about it.

In this table the rows that are related to each other are separated by a row that contains a pipe character e.g.

test line 1_1
test line 1_2
test line 1_3
|
test line 2_1
test line 2_2
test line 2_3
|
test line 3_1
test line 3_2
test line 3_3

etc..

I need to move this into a table and concatenate the lines of each group of rows into one line separated by the pipe delimiter. So from the above example I need to end up with in another table:

test line 1_1 test line 1_2 test line 1_3
test line 2_1 test line 2_2 test line 2_3
test line 3_1 test line 3_2 test line 3_3

I am not sure how to do it. I thought about exporting to a text file then reimporting using the pipe delimiter – this would work but I will be dealing with millions of rows and so I would rather aviod this overhead, there must be a way to do it in SSIS but at the moment I am not sure how.

Any help much appreciated, thanks.

Answer 1

You can create a Data Flow task that uses a Source component. The source will retrieve the data with a SQL statement.

Then, add a Script component as a transformation. The script component must be set as asynchronous. Select the Inputs and Outputs tab of the Script component, and then select Output 0. Set the SynchronousInputID to None. This will allow the component to output a different number of rows than the number of rows input. Add a new column below the Output Columns branch.

In your script component, you can concatenate the data as you wish. By default, your input buffer will be Row and the output buffer will be Output0Buffer. You can use Row.NextRow() to get the next input row and Output0Buffer.AddRow() to add new output rows.

Given your sample data, you can add input data to the output buffer until a ‘|’ appears. Or, you can use a counter if that fits your data.