Home/ Questions/Q 7552571
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-30T10:47:26+00:00

At the moment I’m working on a quite tricky transferring from a .csv file

  • 0

At the moment I’m working on a quite tricky transferring from a .csv file to DB. I have to develop a package/solution/xxxyyy that handles a flow of data from this .csv file to my SQL Server DB (the .csv is updated with new data everyday).

The approach that my boss “suggested” I should use is through SSIS (normally I would have wrote some kind of “parser” to easily convoy the data from the .csv). The fact is that I have quite a bit of transformation to do.

i.e.

An employee has this fields:

name;surname;id;roles

The field “roles” is formatted like this:

role1,role2,role3

This relationship in my db is mapped in 3 different tables:

tblEmployee
    PK_Emp | name | surname

tblRoles
    PK_Role | roleName

tblEmployeeRole
    PK_Emp | PK_Role

So, from the .csv I have to extract the roles of a single employee, insert those in tblRoles (checking that there’s no duplicate). Then I have to manage the relationship in tblEmployeeRole.

Considering that this is just an example of one of the different transformations that I have to manage I was wondering if SSIS is the best tool to achieve my goal (loads if script components). When I explained my perplexities to my boss he came up with this “idea”:

Use SSIS to transfer the data, as they are, in a temporary table then handle the different transformations through stored procedures.

From the very little I know about stored procedure, I’m not sure that I should follow this idea.
Now, considering that my actual superior isn’t that enlightened project manager (he usually mess up our work with bizarre ideas) and considering the fact that I’m not such an expert neither in SSIS nor in stored procedure, I’ve decided to write here and see if anyone can explain me if one of the previous approaches is the right one or if I have to consider some other (better) solution.

Sorry for my poor English, ty for any help =)

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    I would insert the data from the CSV file as-is.
    Then do any parsing in the database end. If this is something that has to be done often I would then take any scripts you have made to do this and create procedures/functions from that. This question is a bit grand-scheme so this is only a general solution. If you need help doing the parsing of the roles into the look up tables then that would be more specific and of better use.
    In general when I work with massive flat-file data sets that need to be parsed into a SQL structure:

    • Import the data as-is
    • Find the commonalities among the look up codes
    • Create the base look up tables (in your case it would be tblRoles)
    • Create a script to insert into both tblEmployee and tblEmployee role
    • Once my test scenarios work then I worry about combining each component step into one monolithic SSIS or stored procedure.

    I suggest something similar here. Break this import task into small pieces and worry about the grand design later. SSIS, procs, compiled code…any of these might work for you. You just need to know what you need it to do.

    • 0