Question

Well, as the title says. I’d like to use a script component destination, and then utilize LINQ to select which rows to process for output.

For a bit more background, I have this ugly merged thing with a one-to-many relationship. The rows look sort of like:

[ID] [Title]   [OneToManyDataID]
1    Item one   2
1    Item one   4
1    Item one   3
3    Item two   1
3    Item two   5

We’ll call the objects [Item], which has the ID and Title columns and [OneToMany]

I was hoping I could throw the entire thing to a script component destination, and then use LINQ to do something like group by the item and only take the data from the highest OneToMany object. Sort of like:

foreach(var item  in Data.GroupBy(d=>d.Item).Select(d=> new {Item = d.Key})){
     //Then pick out the highest OneToMany ID for that row to use with it.
}

I realize there are probably better LINQ queries to accomplish this, but the point is, the script component in SSIS seems to only allow working with it on a per-row basis, with the predefined ProcessInputRow-method. Where I’d like to determine exactly which rows are processed and what properties are passed to that method.

How would I go about doing this?

Answer 1

To restate your problem, how can I make an Script Transformation stop processing row-by-row? By default, a script transformation is going to be a synchronous component – 1 row in, 1 row out. You’ll want to change that to an asynchronous component 1 row in – 0 to many rows out.

On your Script Transformation Editor, the Inputs and Outputs tab, for your output collection Output 0 change the value of SynchronousInputID from whatever it is to None.

Don’t cast stones on my LINQ code-I trust you can handle making that work right. The intention of this code block is to demonstrate how you would collect your rows for processing and then pass them on to a downstream consumer after modifying them. I commented on the methods to help you understand what each one of them does in the script component life cycle but if you’d rather read MSDN they know a bit more than I do 😉

using System;
using System.Data;
using System.Linq;
using System.Collections.Generic;
using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
using Microsoft.SqlServer.Dts.Runtime.Wrapper;

[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
public class ScriptMain : UserComponent
{
    /// <summary>
    /// Our LINQ-able thing.
    /// </summary>
    List<Data> data;

    /// <summary>
    /// Do our preexecute tasks, in particular, we will instantiate
    /// our collection.
    /// </summary>
    public override void PreExecute()
    {
        base.PreExecute();
        this.data = new List<Data>();
    }

    /// <summary>
    /// This method is called once the last row has hit.
    /// Since we will can only find the highest OneToManyDataId
    /// after receiving all the rows, this the only time we can
    /// send rows to the output buffer.
    /// </summary>
    public override void FinishOutputs()
    {
        base.FinishOutputs();
        CreateNewOutputRows();
    }

    /// <summary>
    /// Accumulate all the input rows into an internal LINQ-able
    /// collection
    /// </summary>
    /// <param name="Row">The buffer holding the current row</param>
    public override void Input0_ProcessInputRow(Input0Buffer Row)
    {
        // there is probably a more graceful mechanism of spinning
        // up this struct.
        // You must also worry about fields that have null types.
        Data d = new Data();
        d.ID = Row.ID;
        d.Title = Row.Title;
        d.OneToManyId = Row.OneToManyDataID;            
        this.data.Add(d);
    }

    /// <summary>
    /// This is the process to generate new rows. As we only want to
    /// generate rows once all the rows have arrived, only call this
    /// at the point our internal collection has accumulated all the
    /// input rows.
    /// </summary>
    public override void CreateNewOutputRows()
    {
        foreach (var item in this.data.GroupBy(d => d.ID).Select(d => new { Item = d.Key }))
        {
            //Then pick out the highest OneToMany ID for that row to use with it.
            // Magic happens
            // I don't "get" LINQ so I can't implement the poster's action
            int id = 0;
            int maxOneToManyID = 2;
            string title = string.Empty;
            id = item.Item;
            Output0Buffer.AddRow();
            Output0Buffer.ID = id;
            Output0Buffer.OneToManyDataID = maxOneToManyID;
            Output0Buffer.Title = title;
        }
    }

}
/// <summary>
/// I think this works well enough to demo
/// </summary>
public struct Data
{
    public int ID { get; set; }
    public string Title { get; set; }
    public int OneToManyId { get; set; }
}

Configuration of the Script Transformation