Basically have many huge delimited files that I know I can import as a table, but I need to map that data to an existing rational multi-table MySQL database. There should not be any conflict with datatypes, but I’m super new to this, so please point out anything I should be watching for. Clearly I’m not going to run this in production either until I know it works.
Not 100% sure stackoverflow is the right place to ask a database question, but I couldn’t find any other Stack Exchange that was a better fit.
Posted this question on SuperUser looking for a GUI to do this, but I up for coding this is it gets the job done. As such there is no target language, just the requirement that the database be MySQL.
Also, found this stackoverflow Q/A that deals with MS-SQL’s SSIS (which I’m not planning on using due to cost, but the content and issues faced are of the same nature it appears.) —
Loading Multiple Tables using SSIS keeping foreign key relationships
I’d suggest using the ETL(extract translate load) tool from the Pentaho Business Intelligence package. It’s got a bit of a learning curve but it’ll do exactly what you’re looking for. Their ETL tool is called Kettle and it’s extremely powerful once you get the hang of it.
There are two versions of Pentaho, an enterprise version that has a free trial, and a free community version. The community version is more than capable but you might give the enterprise version a test ride too.
Here’s some links
Pentaho Community Edition Site
Kettle Site
Pentaho Enterprise Site
Update: Multiple table outputs
One of the key steps in your transformation is going to be a combination lookup-update. This step checks a given table to see if a record from your data-stream exists and inserts a new record if it does not. Regardless of whether it’s a new or old record it’s going to append the key field from that record into your data-stream. As you keep going you’ll use these keys as foreign keys as you import data into related tables.