Question

I am developing a dimension table, the requiremnet is to insert all possible unique combinations of source attribute.

step1: I took source with row_number() functions to get the duplicate records as row number

step2: conditional split used to take onlt row == 1 so I am taking all uniqe possible records from source in the initial load also I am considering only unique records.

step3: Lookup on all the columns from source to target dim table. each time lookup with the target table for combinations exists in target dim table

step4: if not matched then insert the record into dim table.

problem: source: col1 col2 col3

              'x'     'x'    null

target exists: col1 col2 col3

              'x'      'x'     null

I took Not matched records path to target table, I am expecting not to get this record from lookup but getting this record out from lookup.

the lookup component is still passing the row to the target so duplicate is forming in the target dim table.

does lookup don’t compare the col3 null value comparison to avoid this record insert.

Could give any info on how to avoid the null comparison in lookup.

thanks

prav

Answer 1

I suggest you use a lookup to identify the NULL and (space) values.