Question

I have data on a WebSphere MQ queue. I’ve written a script task to read the data, and I can output it to a variable or a text file. But I want to use that as input to a dataflow step and transform the data. The ultimate destination is a flat file.

Is there a way to read the variable as a source into a dataflow step? I could write the MQ data to a text file, and read the text file in the dataflow, but that seems like a lot of overhead. Or I could skip the dataflow altogther, and write all the transformations in a script (but then why bother with SSIS in the first place?)

Is there a way to write a Raw File out of the script step, to pass into the dataflow component?

Any ideas appreciated!

Answer 1

If you’ve got the script that consumes the webservice, you can skip all the intermediary outputs and simply use it as a source in your dataflow.

Drag a Data Flow Task onto the canvas and the add a Script Component. Instead of selecting Transformation (last option), select Source.

Double-Click on the Script Component and choose the Input and Output Properties. Under Output 0, select Output Columns and click Add Column for however many columns the web service has. Name them appropriately and be certain to correctly define their metadata.

Once the columns are defined, click back to the Script tab, select your language and edit the script. Take all of your existing code that could write that consumes the service and we’ll use it here.

In the CreateNewOutputRows method, you will need to iterate through the results of the Websphere MQ request. For each row that is returned, you would apply the following pattern.

public override void CreateNewOutputRows()
{
    // TODO: Add code here or in the PreExecute to fill the iterable object, mqcollection

    foreach (var row in mqcollection)
    {
        // Adds a new row into the downstream buffer
        Output0Buffer.AddRow();

        // Assign all the data to the correct locations
        Output0Buffer.Column = row.Column;
        Output0Buffer.Column1 = row.Column1;

        // handle nulls appropriately
        if (string.IsNullOrEmpty(row.Column2))
        {
            Output0Buffer.Column2_IsNull = true;
        }
        else
        {
            Output0Buffer.Column2 = row.Column2;
        }
    }
}

You must handle nulls via the _IsNull attribute or your script will blow up. It’s tedious work versus a normal source but you’ll be far more efficient, faster and consume fewer resources than dumping to disk or some other staging mechanism.