Question

I have two OLEDB Data Sources that have similar columns:

TMP_CRUZTRANS
-------------
CUENTA_CTE numeric (20,0)

TMP_CTACTE_S_USD
----------------
CON_OPE numeric(20,0)

I need to substract all the similar values between this two tables and keep the rows which are different. Is there a transformation/task within SSIS that can perform NOT IN constraint normally used in SQL query?

Currently, I am performing this operation using Execute SQL Task on Control Flow.

The top Data Flow creates the first table TMP_CRUZTRANS (Merge join between other 2 tables… But I guess that’s not important for my question) that i need to keep the different values with the second table.

In the Execute SQL Task, I have the following statement:

INSERT  INTO [dbo].[TMP_CYA] 
SELECT RUT_CLIE, CUENTA_CTE, MONTO_TRANSAC 
FROM [dbo].[TMP_CRUZTRANS]
WHERE CUENTA_CTE NOT IN (SELECT CON_OPE FROM TMP_CTACTE_S_USD)

Finally, with the new table TMP_CYA I can continue with my work.

The problem with this approach is that the TMP_CRUZTRANS got like 5 millions of rows, so it’s VERY slow inserting all this data into a table using Execute SQL Task. It takes about like 5 hours to perform this operation. That’s why I need to do this inside the Data Flow task.

Answer 1

You can use Lookup transformation available within Data Flow task to achieve your requirement.

Here is a sample that illustrates what you are trying to achieve.

Create a package with data flow task. Inside the data flow task, use OLE DB Source to read data from your source table TMP_CRUZTRANS. Use Lookup transformation to validate the existence of the values against the table dbo.TMP_CTACTE_S_USD between given columns. Then redirect the non-matching output to OLE DB Destination to insert rows into table dbo.TMP_CYA

Here is how data flow task would look like in place of the Execute SQL Task that you are currently using.

Configure the Lookup transformation as shown below:

On the General tab page, select Redirect rows to no match output from Specify how to handle rows with no matching entries because you are interested only in non matching rows.

On the Connection tab page, select the appropriate OLE DB Connection manager and select the table dbo.TMP_CTACTE_S_USD. That is the table against which you would like to validate the data.

On the Columns tab page, drag the column CUENTA_CTE and drop it on CON_OPE to establish the mapping between source and lookup tables. Click OK.

When you connect the Lookup transformation with OLE DB Destination, Input Output Selection dialog will appear. Please make sure to select Lookup No Match Output.

Here is the sample before executing the package.

You can see that only 2 rows non matching rows have been transferred to OLE DB destination.

You can notice that the destination table now contains the two non matching rows after package execution.

Hope that helps.