Question

I’m developing a web application that uses some data from a remote database where I have read-only rights. The data in this database, SATMANAGE, is updated every 5th minute. For my application I need some of the columns in one of the remote database tables, called IDIRECT_STATUS.

Atm. I have a cron job performing a query from the CLI and storing the data in a .txt file. It then truncate the local table and uses LOAD DATA LOCAL INFILE to repopulate the updated data.

This has worked out fine up until now. As the application is getting more and more complex, I need a way of altering the local ‘remotes’ table. While I now truncates the local ‘remotes’ table. I need to only update the table with the updated data from the remote ‘IDIRECT_STATUS’ table. This gives me the ability to have columns in the local ‘remotes’ table that don’t exist in the remote ‘IDIRECT_STATUS’ table.

This is what I’m looking for. on the left side is the remote table. I want to keep the right side, local database, updated with the left side, remote database, on the corresponding columns. This without truncating the local table so that the additional local columns keeps there state.

IDIRECT_STATUS                  remotes
-------------------------------|-------------------------------
id                             | id
netmodem_name                  | netmodem_name
nms_name                       | nms_name
ip_addr                        | ip_addr
serial_no                      | serial_no
last_date_online               | last_date_online
last_updated                   | last_updated
network_name                   | network_name
network_id                     | network_id
is_mobile                      | is_mobile
latitude                       | latitude
longitude                      | longitude
lnb_name                       | lnb_name
buc_name                       | buc_name
upstream_qos_profile_name      | upstream_qos_profile_name
downstream_qos_profile_name    | downstream_qos_profile_name
inroute_maximum_data_rate      | inroute_maximum_data_rate
outroute_maximum_data_rate     | outroute_maximum_data_rate
upstream_cir                   | upstream_cir
use_upstream_cir               | use_upstream_cir
crtp_enabled                   | crtp_enabled
INROUTE_GROUP_ID               | INROUTE_GROUP_ID
                               | excluded           (Not updated or deleted)
                               | excluded_to        (Not updated or deleted)
                               | shipowner_id       (Not updated or deleted)
                               | etc...             (Not updated or deleted ...)

Hopefully you awesome guys are able to help me. I anything is unclear please let me know and I’ll edit the question asap!

Answer 1

I saw a field called last_updated in the left table. What I would suggest is to use a simple perl or php script to do the loading that would do the following:

a) theck the newest update time from the local table
b) select data from the remote table that has been update after the last insert
c) update the local table with remote data

in the php code it would look something like this:

<?
// select the last update time
$sql = "select max(last_updated) as last_update_time from remotes";
$r = mysql_query($sql);
$row = mysql_fetch_assoc($r);
// save the last upate time
$last_update_time = $row['last_update_time'];

// get newer rows from the remote database
$sql = "select * from IDIRECT_STATUS where last_updated > ".$last_update_time;
$r = mysql_query($sql);
while ($row = mysql_fetch_assoc($r)) {
  // generate update queries
  $sql = "update remotes set netmodem_name='".$row['netmodem_name']."', ... last_updated='".$row['last_updated']."' where id=".$row['id'];
  // execute query
  mysql_query($sql);
}
?>