Question

tl;dr

What is the EzAPI code to use an OLE DB Source with data access mode of “SQL command from variable” and assign a variable?

Preamble

Once a month, we need to refresh our public test site with subsets of production data. We have determined that for our needs, an SSIS solution provides the best fit for accomplishing this task.

My goal is to systematically build a large number (100+) of “replication” packages. EzAPI is a friendly wrapper to the SSIS object model and it seems like a great way to save mouse-clicks.

I would like for my packages to look like

• Variable – “tableName”; [Schema].[TableName]
• Variable – “sourceQuery”; SELECT * FROM [Schema].[TableName]
• DataFlow – “Replicate Schema_TableName”
• • OLE DB Source – “Src Schema_TableName”; Data Access Mode: SQL command from variable; Variable name: User::sourceQuery
• • OLE DB Destination – “Dest Schema_TableName”; Table or view name variable- fast load; Variable name – User::tableName

Code

This is the code for my table to table replication package.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Microsoft.SqlServer.SSIS.EzAPI;
using Microsoft.SqlServer.Dts.Runtime;

namespace EzApiDemo
{
    public class TableToTable : EzSrcDestPackage<EzOleDbSource, EzSqlOleDbCM, EzOleDbDestination, EzSqlOleDbCM>
    {
        public TableToTable(Package p) : base(p) { }

        public static implicit operator TableToTable(Package p) { return new TableToTable(p); }


        public TableToTable(string sourceServer, string database, string table, string destinationServer) : base()
        {
            string saniName = TableToTable.SanitizeName(table);
            string sourceQuery = string.Format("SELECT D.* FROM {0} D", table);

            // Define package variables
            this.Variables.Add("sourceQuery", false, "User", sourceQuery);
            this.Variables.Add("tableName", false, "User", table);

            // Configure DataFlow properties
            this.DataFlow.Name = "Replicate " + saniName;
            this.DataFlow.Description = "Scripted replication";

            // Connection manager configuration
            this.SrcConn.SetConnectionString(sourceServer, database);
            this.SrcConn.Name = "PROD";
            this.SrcConn.Description = string.Empty;

            this.DestConn.SetConnectionString(destinationServer, database);
            this.DestConn.Name = "PREPROD";
            this.DestConn.Description = string.Empty;

            // Configure Dataflow's Source properties
            this.Source.Name = "Src " + saniName;
            this.Source.Description = string.Empty;
            this.Source.SqlCommand = sourceQuery;

            // Configure Dataflow's Destination properties
            this.Dest.Name = "Dest " + saniName;
            this.Dest.Description = string.Empty;
            this.Dest.Table = table;
            this.Dest.FastLoadKeepIdentity = true;
            this.Dest.FastLoadKeepNulls = true;
            this.Dest.DataSourceVariable = this.Variables["tableName"].QualifiedName;
            this.Dest.AccessMode = AccessMode.AM_OPENROWSET_FASTLOAD_VARIABLE;
            this.Dest.LinkAllInputsToOutputs();
        }

        /// <summary>
        /// Sanitize a name so that it is valid for SSIS objects. 
        /// Strips []/\:=
        /// Replaces . with _
        /// </summary>
        /// <param name="name"></param>
        /// <returns></returns>
        public static string SanitizeName(string name)
        {
            string saniName = name.Replace("[", String.Empty).Replace("]", string.Empty).Replace(".", "_").Replace("/", string.Empty).Replace("\\", string.Empty).Replace(":", string.Empty);
            return saniName;
        }
    }
}

Invocation looks like TableToTable s2 = new TableToTable(@"localhost\localsqla", "AdventureWorks", "[HumanResources].[Department]", @"localhost\localsqlb"); and that builds a package that does what I want except for using a variable in the source.

Problem

The above code supplies the access mode as SQL Query and the query is embedded in the OLE Source. The desire it to use “SQL Command From Variable” and that variable being @[User::sourceQuery] What I’m stuck on is using a variable in the source.

It should be a simple matter of assigning something like

        this.Source.DataSourceVariable = this.Variables["sourceQuery"].QualifiedName;
        this.Source.AccessMode = AccessMode.AM_SQLCOMMAND_VARIABLE;

This results in the correct data access mode selected but the variable isn’t populated.

You can observe that I perform a similar step in the destination which does accept the variable and works “right.”

        this.Dest.DataSourceVariable = this.Variables["tableName"].QualifiedName;
        this.Dest.AccessMode = AccessMode.AM_OPENROWSET_FASTLOAD_VARIABLE;

What doesn’t work

Listing out the permutations I’ve attempted

        this.Source.AccessMode = AccessMode.AM_OPENROWSET;

Results in Data Access Mode set to Table or View and name of table or the view is blank.

        this.Source.AccessMode = AccessMode.AM_OPENROWSET_VARIABLE;

Results in Data Access Mode set to “Table or view name variable” and variable name is sourceQuery. Very close to what I want, except the access mode is not correct. Were this package to run, it’d blow up as the OpenRowSet would expect a straight table name.

        this.Source.AccessMode = AccessMode.AM_SQLCOMMAND;

Results in Data Access Mode set to “SQL Command” and the SQL command text is “User::sourceQuery” That’s the literal value of the variable name so it’s the right thing but since the access mode is wrong, it doesn’t work.

        this.Source.AccessMode = AccessMode.AM_OPENROWSET_FASTLOAD;
        this.Source.AccessMode = AccessMode.AM_OPENROWSET_FASTLOAD_VARIABLE;

Niether of these are correct access modes as they are for destinations (I still tried them but they didn’t work as expected).

At this point, I thought I’d try to work backwards by creating a package that has the OLE DB source defined as I want it and then inspect the source object’s properties.

        Application app = new Application();
        Package p = app.LoadPackage(@"C:\sandbox\SSISHackAndSlash\SSISHackAndSlash\EzApiPackage.dtsx", null);
        TableToTable to = new TableToTable(p);

My code has set both SqlCommand and DataSourceVarible with the variable’s qualified name. I’ve pulled down changeset 65381 and compiled that (after fixing some references to the SQL Server 2012 dlls) in hopes there might have been a fix since the Dec 30 2008 Stable build but to no avail.

Have I found a bug in their code or am I just missing something?

Answer 1

The current, stable build of EzAPI does not support the assignment of a variable as an OleDB Source property. I opened a similar discussion over on CodePlex and ended up learning more about how all of this works.

The root problem is the related property “SqlCommandVariable” should be set when the access mode is set to “SQL Command from Variable.” Currently, the code only covers destination variables.

My resolution was to download the source code and modify the setter for the property DataSourceVariable in EzComponents.cs (line 1027 for changeset 65381)

        set 
        { 
            m_comp.SetComponentProperty("OpenRowsetVariable", value); 
            if (AccessMode == AccessMode.AM_SQLCOMMAND_VARIABLE)
            {
                m_comp.SetComponentProperty("SqlCommandVariable", value); 
            }
            ReinitializeMetaData(); 
        } 

If you’re looking to get this problem resolved properly, you may upvote the Issue