Question

I am working on a mysql query for a report. The idea is to have a simple table say ‘reportTable’ with the values being fetched from various places. I could then use the reportTable more easily without remembering lots of joins etc and also share this table for other projects.

1. Should I break down the inner insert part of the query so it does
   chunks at a time I will be adding probably tens of thousands of rows?

---

INSERT INTO reportTable
(
   -- long query grabbing results from various places
   SELECT var1 FROM schema1.table1
   SELECT var2 FROM schema2.table1
   SELECT var2 FROM schema2.table1
   etc
)

Answer 1

This addresses your concerns that inserting data takes too long and so on. I understood it like you rebuild your table each time. So, instead of doing so, just fetch the data that is new and not already in your table. Since looking up if the data is already present in your report table might be expensive, too, just get the delta. Here’s how:

Make sure that in every table you need a column like this is present:

ALTER TABLE yourTable ADD COLUMN created timestamp DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP;

The ON UPDATE clause is of course optionally, don’t know if you need to keep track of changes. If so, give me a comment and I can provide you with a solution with which you can keep a history of your data.

Now you need a small table that holds some meta information.

CREATE TABLE deltameta (tablename varchar(50), LSET timestamp, CET timestamp);

LSET is short for Last Successful Extraction Time, CET for Current Extraction Time.

When you get your data it works like this:

UPDATE deltameta SET CET = CURRENT_TIMESTAMP WHERE tablename = 'theTableFromWhichYouGetData';
SELECT @varLSET := LSET, @varCET := CET FROM deltameta WHERE tablename = 'theTableFromWhichYouGetData';
INSERT INTO yourReportTable (
    SELECT whatever FROM aTable WHERE created >= @varLSET AND created < @varCET
);
UPDATE deltameta SET LSET = CET WHERE tablename = 'theTableFromWhichYouGetData';

When anything goes wrong during inserting your script stops and you get the same data the next time you run it. Additionally you can work with transactions here, if you need to roll back. Again, write a comment if you need help with this.