Question

I have a large data set in a denormalized format. Here is an example of the column names:

foreign_key_ID, P1, P2, P3, P4, P5…. D1, D2, D3…. etc..

These fields all contain similar type of data.

I need to normalize this into my existing table structure:

insert into new_table (id, name, index)
select foreign_key_id, P1, 1
from denormalized_table;

But that means that I need to run separate queries for each field in my denormalized table, just changing a few things:

insert into new_table (id, name, index)
select foreign_key_id, P2, 2
from denormalized_table;

This is getting tedious considering how many of these fields I have.

Is there a way this can be automated into a single operation? I.e.: iterate through the fields (I don’t mind creating a list of eligible fields once, somewhere), pull off the last digit of that field name (ie “1” in “P1” and “2” for “P2”) use the field name and the extracted index # in the sub-select.

Answer 1

Here’s a start:

SELECT column_name, substr(column_name,2) AS `index`
  FROM information_schema.columns
 WHERE table_schema = 'mydatabasename'
   AND table_name = 'denormalized_table'
   AND column_name REGEXP '^[PD][0-9]+$'
 ORDER BY column_name

You can modify the select list in that statement, to have MySQL generate statements for you:

SELECT CONCAT('INSERT INTO new_table (id, name, `index`) SELECT foreign_key_id, '
         ,column_name,', ',substr(column_name,2)
         ,' FROM denormalized_table ;') AS stmt
  FROM information_schema.columns
 WHERE table_schema = 'mydatabasename'
   AND table_name = 'denormalized_table'
   AND column_name REGEXP '^[PD][0-9]+$'
 ORDER BY column_name

The output from that would be a set of MySQL INSERT statements that you could then execute.

---

If the number of rows and total size of the data to be inserted is not too large, you could and you want to get the whole conversion done in “one operation”, then you could generate a single INSERT INTO … SELECT statement, using the UNION ALL operator. I would get the majority of the statement like this:

SELECT CONCAT('UNION ALL SELECT foreign_key_id, '
         ,column_name,', ',substr(column_name,2)
         ,' FROM denormalized_table ') AS stmt
  FROM information_schema.columns
 WHERE table_schema = 'mydatabasename'
   AND table_name = 'denormalized_table'
   AND column_name REGEXP '^[PD][0-9]+$'
 ORDER BY column_name

I would take the output from that, and replace the very first UNION ALL with an INSERT INTO .... That would give me a single statement to run to get the whole conversion done.