Question

Into:

Currently i have scraped all the data into one PostgreSQL ‘Bigtable’ table(there are about 1.2M rows). Now i need to split the design into separate tables which all have dependency on the Bigtable. Some of the tables might have subtables. The model looks pretty much like snowflake.

Problem:

What would be best option to inserting data into tables? I thought to make the insertion with functions written in ‘SQL’ or PLgSQL. But the problem is still with auto-generated ID-s.

Also if you know what tools might make this problem solving easier then post!

//Edit i have added example, this not the real case just for illustration

Answer 1

1.2 M rows is not too much. The best tool is sql script executed from console “psql”. If you have a some newer version of Pg, then you can use inline functions (DO statement) when it is necessary. But probably the most useful command is INSERT INTO SELECT statement.

-- file conversion.sql
DROP TABLE IF EXISTS f1 CASCADE;
CREATE TABLE f1(a int, b int);
INSERT INTO f1
   SELECT x1, y1 
      FROM data
     WHERE x1 = 10;

    ...

-- end file

psql mydb -f conversion.sql