Question

I am a bit rusty with mysql and trying to jump in again..So sorry if this is too easy of a question.

I basically created a data model that has a table called “Master” with required fields of a name and an IDcode and a then a “Details” table with a foreign key of IDcode.

Now here’s where its getting tricky..I am entering:

INSERT INTO Details (Name, UpdateDate) Values (name, updateDate)

I get an error: saying IDcode on details doesn’t have a default value..so I add one then it complains that Field ‘Master_IDcode’ doesn’t have a default value

It all makes sense but I’m wondering if there’s any easy way to do what I am trying to do. I want to add data into details and if no IDcode exists, I want to add an entry into the master table. The problem is I have to first add the name to the fund Master..wait for a unique ID to be generated(for IDcode) then figure that out and add it to my query when I enter the master data. As you can imagine the queries are going to probably get quite long since I have many tables.

Is there an easier way? where everytime I add something it searches by name if a foreign key exists and if not it adds it on all the tables that its linked to? Is there a standard way people do this? I can’t imagine with all the complex databases out there people have not figured out a more easier way.

Sorry if this question doesn’t make sense. I can add more information if needed.

p.s. this maybe a different question but I have heard of Django for python and that it helps creates queries..would it help my situation?

Thanks so much in advance 🙂

Answer 1

(decided to expand on the comments above and put it into an answer)

I suggest creating a set of staging tables in your database (one for each data set/file).

Then use LOAD DATA INFILE (or insert the rows in batches) into those staging tables.
Make sure you drop indexes before the load, and re-create what you need after the data is loaded.

You can then make a single pass over the staging table to create the missing master records. For example, let’s say that one of your staging table contains a country code that should be used as a masterID. You could add the master record by doing something along the lines of:

insert 
  into master_table(country_code)
select distinct s.country_code 
  from staging_table     s
  left join master_table m on(s.country_code = m.country_code)
 where m.country_code is null;

Then you can proceed and insert the rows into the “real” tables, knowing that all detail rows references a valid master record.

If you need to get reference information along with the data (such as translating some code) you can do this with a simple join. Also, if you want to filter rows by some other table this is now also very easy.

insert 
  into real_table_x(
          key
         ,colA
         ,colB
         ,colC
         ,computed_column_not_present_in_staging_table
        ,understandableCode
       )
  select x.key
        ,x.colA
        ,x.colB
        ,x.colC
        ,(x.colA + x.colB) / x.colC
        ,c.understandableCode
     from staging_table_x  x 
     join code_translation c on(x.strange_code = c.strange_code);

This approach is a very efficient one and it scales very nicely. Variations of the above are commonly used in the ETL part of data warehouses to load massive amounts of data.

One caveat with MySQL is that it doesn’t support hash joins, which is a join mechanism very suitable to fully join two tables. MySQL uses nested loops instead, which mean that you need to index the join columns very carefully.
InnoDB tables with their clustering feature on the primary key can help to make this a bit more efficient.

One last point. When you have the staging data inside the database, it is easy to add some analysis of the data and put aside “bad” rows in a separate table. You can then inspect the data using SQL instead of wading through csv files in yuor editor.