Question

I am trying to load data from a text file into a MySQL table, by calling MySQL’s LOAD DATA INFILE from a Java process. This file can contain some data for the current date and also for previous days. The table can also contain data for previous dates. The problem is that some of the columns in the file for previous dates might have changed. But I don’t want to update all of these columns but only want the latest values for some of the columns.

Example,

Table

+----+-------------+------+------+------+
| id | report_date | val1 | val2 | val3 |
+----+-------------+------+------+------+  
|  1 | 2012-12-01  |   10 |    1 |    1 |  
|  2 | 2012-12-02  |   20 |    2 |    2 |
|  3 | 2012-12-03  |   30 |    3 |    3 |
+----+-------------+------+------+------+

Data in Input file:

1|2012-12-01|10|1|1
2|2012-12-02|40|4|4
3|2012-12-03|40|4|4
4|2012-12-04|40|4|4
5|2012-12-05|50|5|5

Table after the load should look like

mysql> select * from load_infile_tests;
+----+-------------+------+------+------+
| id | report_date | val1 | val2 | val3 |
+----+-------------+------+------+------+
|  1 | 2012-12-01  |   10 |    1 |    1 |
|  2 | 2012-12-02  |   40 |    4 |    2 |
|  3 | 2012-12-03  |   40 |    4 |    3 |
|  4 | 2012-12-04  |   40 |    4 |    4 |
|  5 | 2012-12-05  |   50 |    5 |    5 |
+----+-------------+------+------+------+
5 rows in set (0.00 sec)

Note that column val3 values are not updated. Also I need to do this for large files as well, some files can be >300Megs or more, and so it needs to be a scalable solution.

Thanks,
Anirudha

Answer 1

It would be good to use LOAD DATA INFILE with REPLACE option, but in this case records will be dropped and added again, so old val3 values will be lost.

Try to load data into temporary table, then update your table from temp. table using INSERT … SELECT/UPDATE or INSERT … ON DUPLICATE KEY UPDATE statements.