Home/ Questions/Q 8809739
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-14T02:57:07+00:00

Kinda racking my brain here trying to do what I thought would be simple

  • 0

Kinda racking my brain here trying to do what I thought would be simple to do. I’m trying to move to a shell script/MySQL option for getting data from one table in a db to a staging table in another db. Basically truncate table db2.stage_customer_log, then db1.customer_log -> db2.stage_customer_log. I tried a couple options with limited success on each, and I’m ready for suggestions.

The first think I tried was: 

mysqldump -p dbname -u uname customer_log > stage_customer_log.csv --no-create-info

But it turned out that wasn’t a good option since the resulting file was an INSERT into the original table name. So I’d have to do a little manipulation to get it to work.

Next, I created a shell script with this: 

#!/bin/sh
mysql -h hostname -P 99999 -u uname -p --database dbname <<STOP

SELECT * FROM customer_log INTO OUTFILE 'stage_dm_customer_log.csv'
FIELDS TERMINATED BY ',' ENCLOSED BY '"'
LINES TERMINATED BY '\r\n'; 

\q
STOP
test $? = 0 && echo "Your batch job terminated gracefully"

This gave me the result I wanted (comma delimited file), but it put the file in the MySQL directory (even if I entered an absolute path, or ‘./’ or whatever). Seeing Im hosted on Amazon, the second part (the importing) doesn’t work, because I get access errors.

Then I went back to the first option when I read more on the options. My final command was this: 

mysqldump -p dbname -u uname customer_log --no-create-info --no-create-db --tab='/tmp/' --fields-optionally-enclosed-by='"' --fields-terminated-by=',' --fields-escaped-by='' --lines-terminated-by='\n' --verbose

but again ended up with a file name that had to be manipulated.

I finally tried Pentaho to bring in tables, and I don’t know if its our instance but it takes quite a long time for such little records (32 min for 85,000). So, I dont need specifics but what is the best route to go for something like this? What have others implemented?

Thanks.

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    If both of the databases reside on the same host, then you can run this command:

    INSERT INTO db2.stage_customer_log (column1, column2, ...)
SELECT column1, column2, ...
FROM db1.stage_customer_log
WHERE [your filters here]

    But, if they are on different servers, then this might be helpful:

    mysqldump -u user1 -ppassword1 -h host1 [some options here] db1 stage_customer_log | mysql -u user2 -ppassword2 -h host2 [some options here] db2

    mysqldump always saves the csv file on the server computer and sql dump on the client side.

    UPDATE 1

    Another trick, could be either using MySQL Workbench (you could save the dump files on your local hard drive) or using FEDERATED storage engine.

    • 0