Question

I have a very large database (with new information being added everyday) that a little cakePHP app was built for searching the database online. Because it is too big and slow,I want to stage the data in a new database with:

- only the last 12 months 
- converts the serial varchar field data into INT (for sorting and "in btw" searches)
- Only take numeric serials less than 5 numbers (< 99999)

OLD DATABASE

 id(key)           serial (varchar)   Date(datetime)                                           
 --------          -----------    --------------  
 1                 12345              2011-02-15 23:50:26   
 2                 12345678           2008-12-15 23:50:26
 3                 abc45              2009-12-15 23:50:26

NEW DATABASE

 id(key)           serial (INT)   Date(datetime)                   
 --------          -----------    --------------  
 1                 12345          2011-02-15 23:50:26

.
I need a MYSQL script that i can schedule to run everyday and append new entries (with the above conditions) to the new database and use this database for the cakePHP app

Answer 1

You could execute a daily script, after first running an initial import:

First script, imports 12 months of data:

INSERT INTO new_database.table_name (serial, date)
SELECT old.serial, old.date
FROM old_database.table_name old
WHERE old.date > DATE_SUB(CURDATE(), INTERVAL 1 YEAR)
AND LENGTH(old.serial) < 6;

Daily script, just imports the previous 24 hours:

INSERT INTO new_database.table_name (serial, date)
SELECT old.serial, old.date
FROM old_database.table_name old
WHERE old.date > DATE_SUB(CURDATE(), INTERVAL 24 HOUR)
AND LENGTH(old.serial) < 6;

I interpreted your serial number length criteria as 5 characters or less (< 6) as your example has 5 digits.

In case the daily script doesn’t run exactly on time, to avoid skips or duplicates you may want to edit the scripts somewhat. For example, if the serial numbers are unique you can avoid inserting duplicate records by selecting only serials that are not already present in the new database table.