Question

I load files in MySQL. Each file contains:

Date ShopNumber subject price quantity total

e.g. file1:

01/01/2011 Mexico Book1 $10 2 $20
01/01/2011 Mexico Book2 $10 2 $20
01/01/2011 Mexico Book3 $10 2 $20
...............................

file2:

01/01/2011 Mexico Book4 $10 2 $20
01/01/2011 Mexico Book1 $10 1 $10
...............................

In MySQL table:

01/01/2011 Mexico Book1 $10 3 $30
01/01/2011 Mexico Book2 $10 2 $20
01/01/2011 Mexico Book3 $10 2 $20
01/01/2011 Mexico Book4 $10 2 $20
...............................

The problem is: I have to track the source file and I cannot include source file to unique key of table – I need only keep totals.
Duplicates are rare. May be possible add another table that will keep only duplicates? Thanks.

Answer 1

My application includes import functionaly and here is the way I solve the problem of loading data into MySQL and tracking information.

• when the user uploads a file the application re-formats it to the tab-separated uft8-encoded valid temporary file
• the name of the file and some other meta information about the import the application inserts into a separate table import_info and retrieves its id (update_id)

• then the application loads the data from the file using LOAD DATA INFILE query to which it passes the path to the file and the update_id (attr1, attr2, attr3 come from file, update_id comes from parameter to the query)

  LOAD DATA LOCAL INFILE %(file_path)s
  INTO TABLE importdata
  CHARACTER SET 'utf8'
  (attr1, att2, att3, update_id)
  SET update_id = %(update_id)s;
  

• the importdata table in my case allows duplicates which the application merges in other tables after the data is imported. If the merge is successful, I remove the imported data from the importdata table.