Question

I need to use the same Stored Procedures against many tables all with the same structure in my DB. This is data loaded from customers,with one table/customer and the data needs calculations/checks run before it’s loaded to our DataWarehouse.

So far these are the options and issues I’ve found and I’m looking for a better pattern/approach.

1. Create a view that points to the
   table I want to process, the SPs
   then talk to that view. This works
   well (especially once I’d worked out
   how to create views ‘automagically’
   based on their columns). But the
   view can only be used with one table
   at a time, forcing the system to
   deal with one customer at a time.

2. Use dynamic sql within each SP –
   makes the SPs much harder to
   read/debug and for those reasons has
   been ruled out

3. Create a partitioned view across
   all the tables and then use a
   paramatised table function to return
   just the data we’re interested in –
   ah but then I can’t update the data
   as the function returns a table that
   can be only used for select

4. Use dynamic sql inside a function
   (can’t be done) to create a view
   (which also can’t be done) …. give
   up
5. Within the SP create a temp table
   with over the target table using
   dynamic sql, but then the temp table
   only exists in the session that runs
   the dynamic sql not the ‘parent’
   session that’s running the SP …
   give up
6. Create a global temp table using
   dynamic SQL to avoid the scope issue
   of 5, then run the SP against the
   global temp table. Still run into
   the single customer issue.
7. Create the view as in 1 within a
   transaction and then run all the SPs
   and then commit – works fine for one
   user, but any others are now blocked
   trying to create a new view of the
   same name
8. Use a temporary view … can’t in
   T/Sql
9. Move all the code into .Net – but
   we have environment issues where
   tsql is much easier to host/run

I know I’m not the only person who has this problem, have any of you good people solved it, please help.

Answer 1

Maybe your approach is wrong, I will go deep in details in a while but it seems that your problem can be solved using SSIS

— Updated answer:

First, the big picture:

The most affordable way to process the tables dynamically is using a script instead of a stored procedure. If you want to make table access randomly chosen, you certainly will not use any of the performance advantages of stored procedures, i.e. execution plans. A SQL Script can be easily upgraded to point one table at runtime using placeholders and replacing it before executing.

The script can be loaded from the filesystem, a variable, a text column in a table, etc. The loading process consists in read the script content to a string variable. This step occurs once.

The next step is the preparation stage. This step will be executed for each table to be processed. The main business of this step is to replace the table placeholders with the current table being processed. Also is possible to set parameter values like any parameter you can need to pass into the sp that you already wrote.

The last step is the execution of the script. As is already loaded into a variable and the placeholders were set to the current table name, you can safely call a ExecuteSQLTask with the sql variable as the input. This process of course happens for each table you want to process.

Ok. Now let’s see this in action.

This is a sample database model:

CREATE TABLE [dbo].[t_n](
  [id] [int] IDENTITY(1,1) NOT NULL,
  [name] [varchar](50) NOT NULL,
  [start] [datetime] NULL,
  CONSTRAINT [PK_t_n] PRIMARY KEY CLUSTERED ([id] ASC)
) ON [PRIMARY]

where t_n represents any table (t_1, t_2, t_3, etc).

This is your current stored procedure:

CREATE PROCEDURE SpProcessT_n 
AS
BEGIN
    SET NOCOUNT ON;
    SELECT * FROM [t1]; 
END
GO

Now, transform this stored procedure to a Sql script, placing a placeholder instead of the table name

    SET NOCOUNT ON;
    SELECT * FROM [$table_name];

I choose to save this in a .sql file in the filesystem to keep the POC as simple as possible.

Next, create a SSIS Package like this:

These are the settings I choose to set up the loop:

And this is the way you can assign the table name to a variable called appropriately _table_name_

This is the setup of the script task, here you find that the variable _table_name_ has read only access, while a new variable called SqlExec has read/write access:

And this is it’s Main function:

    public void Main()
    {
        String Table_Name = Dts.Variables["table_name"].Value.ToString();
        String SqlScript;
        Regex reg = new Regex(@"\$table_name", RegexOptions.Compiled);
        using (var f = File.OpenText(@"c:\sqlscript.sql")) {
            SqlScript = f.ReadToEnd();
            f.Close();
        }
        SqlScript = reg.Replace(SqlScript, Table_Name);
        Dts.Variables["SqlExec"].Value = SqlScript;
        Dts.TaskResult = (int)ScriptResults.Success;
    }

You can notice that the Dts Variable SqlExec contains the sql script that will be executed. Now you can set the following options in your ExecuteSqlTask:

Successfully tested in MSSQL 2008, if you put a insert inside the script file you will notice new rows in each table.

Hope this helps!