Question

I have a raw table used as a kind of buffer where periodically new data is inserted (average of ~20.000 rows inserted in bulk each ~5 minutes). Then there is a stored PL/SQL proceudre which reads this raw table and insert the information in the distinct tables of the database.
Unfortunately I cannot do a direct bulk insert in my destination table from the rows stored in the raw table, because part of the columns (10 out of 20) are foreign keys in the destination table. This means that before doing the insert I have to resolve all the foreign key id’s and then use these to insert the new rows in the destination table.
Moreover, the element in the foreign key table could not be there, so it needs to be inserted in the foreign key table and then its id can be used as foreign key id.

The solution I am going to use is to have a stored procedure that:
1) for each column of the raw table that corresponds to a foreign key in the destination table I select the distinct values of the column and for each one I select the id in the foreign table, and if it doesn’t exists I insert it reutrning the relative id;
2) I write out to an empty temporary table (truncated each time before inserting into it) the same rows of the raw table but with the resolved foreign keys;
3) I do a bulk insert in the destination table by selecting the values from the temporary table.

Why using a temporary table? Because I want to avoid to insert row by row in a destination table that has more then 50milions of lines.

(I have inerithed the oracle sql schema, and part of the solution structure; I cannot even change it since there is a uge set of monitoring tools that at the moment cannot be changed).

How can I optimize the code by resolving the foreign keys before inserting a new rows in the destination table? Is there any other better scaling solutions to suggests?

I tried to summarize I complex workflow, leaving out some details and generalizing a bit the context.

Answer 1

What percentage of the foreign keys don’t resolve ?

You could do the following for each of those foreign key master tables

INSERT INTO parent_table (id, name)
SELECT id.nextval, r2.name
FROM (SELECT DISTINCT r.name FROM raw)
WHERE r2.name NOT IN (SELECT name FROM parent_table)

Then you will know they all exist and can just do the insert.

Or, if mostly they are there, you can look at BULK.. EXCEPTIONS INTO and let the SQL engine tell you which ones fail and deal with them individually.