Question

When would you use Integration Services and when would just use SQL/stored procedures? What are the advantages of using one or the other?

If you are migrating data from a legacy system, would you use SSIS or just SQL?

Answer 1

MIGRATING DATA FROM LEGACY SYSTEM

1. If you are migrating data from a legacy system and you have the option of restoring the legacy system database to your new server, then you don’t necessarily need SSIS. T-SQL commands on the new server are probably going to be faster to script out and run than a SSIS data flow if you are taking the raw data in whole.

2. If the legacy data is only accessible via linked server or through SSIS, then you are probably better off using SSIS to create the destination tables and load the data. Stored procedure calls that pull the data across via linked server are generally slower than SSIS data flow connections.

3. If the legacy server is a non-SQL Server database (like MYSQL or Oracle), then you are probably best off using SSIS and data flows. You could do this via linked servers, but it will probably require more work.

4. Even if you choose to use SSIS to pull data from another server, I would recommend creating all the tables as T-SQL commands rather than relying on the create table option in the Destination data flow. The create table option doesn’t create indexes, clustered indexes, primary keys, etc. and it also defaults every field to NULL. When migrating data from a legacy server, I usually script out the original object, modify the script, and run the modified script on my target server.

5. I would not recommend using the default wizards in SSIS to import data unless it is a one-time process. These are generally not the best from either a performance stand-point or from a maintenance stand-point. Even when loading the data once, I generally prefer to script out each object individually to make certain I don’t repeat the errors of the legacy database in the new database.

6. If you intend to load the data from the legacy system in a reoccurring manner, then I would recommend using a SSIS package just for maintainability and extensibility. Even if you decide it needs to execute nothing but stored procedures, at least you can parallelize it and organize the stored procedure calls in a meaningfull way. If you go with just executing the stored procedure sequentially in a SQL Server Agent job step, then you aren’t going to be able to easily parallelize the process.

OTHER SCENARIOS

1. I generally call stored procedures as SQL Server Agent job steps when I want to segregate processes into individual steps either for recoverability or performance monitoring purposes. For instance, if I know I need to prepare data before kicking off a SSRS report job, then I usually will make a two-step SQL Server agent job. The first loads data into a table and the second kicks off a shared schedule.

2. Another scenario is that I have several core data warehouse tables that need to be loaded before I run several other dependent processes. I will usually use a stored procedure to handle complex scrubbing rules that load a table and then kick off several other jobs that use the newly loaded table. All the subsequent steps can either be T-SQL commands to kick off other jobs, T-SQL commands to scrub more data, SSIS packages to move data between servers or output files, etc.

3. I always use SSIS to import data or export data. I do not use equivalent T-SQL commands to do this even though they are available. I do this because I want the logging, recovery options, ease of development, ease of maintenance, and standardized package workflows.

4. Since I run a team of BI developers who do anything from writing complex T-SQL statements to developing SSIS packages or SSRS reports, SSIS packages give us an easy way for anyone on my team to understand other team members’ work.