Question

I want to load the data from text file to database, if data already exists i need to escape that data while loading.

I am using below query to load the data from text file to mysql data base.

“Load data infile ‘F:/wbrdata.txt’ into table wbrdatatable
fields terminated by ‘,’
optionally enclosed by “”
lines terminated by ‘\r\n’
Ignore 1 lines (channel, time, pulserate, dwellid, targetid);”

It is appending the data to existing table data. I want to avoid the common data which is already exists(duplicates) in table & file while loading to the database.

How can i achieve this?
thank you

regards
sankar

Answer 1

Try to load text file into the temporary table (the same as target table), then remove duplicates from the temporary table and copy the rest to the target table.

---

Example (suppose that wbrdatatable_temp is a temporary table with all data from text file):

CREATE TABLE wbrdatatable(
  id INT(11) NOT NULL AUTO_INCREMENT,
  column1 VARCHAR(255) DEFAULT NULL,
  PRIMARY KEY (id)
);

INSERT INTO wbrdatatable VALUES 
  (1, '111'),
  (2, '222'),
  (3, '333'),
  (4, '444'),
  (5, '555');

CREATE TABLE wbrdatatable_temp(
  id INT(11) NOT NULL AUTO_INCREMENT,
  column1 VARCHAR(255) DEFAULT NULL,
  PRIMARY KEY (id)
);

INSERT INTO wbrdatatable_temp VALUES 
  (1, '111'),
  (2, '222'),
  (10, '100'), -- new record that should be added
  (11, '200'); -- new record that should be added

-- Copy only new records!
INSERT INTO wbrdatatable
SELECT t1.* FROM wbrdatatable_temp t1
  LEFT JOIN wbrdatatable t2
    ON t1.id = t2.id AND t1.column1 = t2.column1
WHERE t2.id IS NULL;

-- Test result
SELECT * FROM wbrdatatable;
+----+---------+
| id | column1 |
+----+---------+
|  1 | 111     |
|  2 | 222     |
|  3 | 333     |
|  4 | 444     |
|  5 | 555     |
| 10 | 100     | -- only new record is added
| 11 | 200     | -- only new record is added
+----+---------+