Question

All,

I am trying to bulk insert some data in a table using the COPY TO command and I can’t seem to get around the unique key error. Here’s my workflow.

Create a dump of the data I want to move to another server

COPY (
  SELECT *
  FROM mytable
  WHERE created_at >= '2012-10-01')
    TO 'D:\tmp\file.txt'

Create a new “temp” table in the target DB then COPY the data like so.

COPY temp FROM 'D:\tmp\file.txt'

I now want to move the data from the “temp” table in to the master table in the target DBlike so.

 INSERT INTO master SELECT * FROM temp 
                WHERE id NOT IN (SELECT id FROM master)

This runs fine but nothing gets inserted and no fields are updated. Does anyone have a clue what might be going on here? The schemas for temp and master are identical. Any help on this matter would be great! I am using Postgresql 9.2

Adam

Answer 1

This can happen if there’s a null value in the IN list.

In SQL, the presence of a null when making comparisons is always false (you need the special IN NULL test to get a match). This has the unfortunate consequence of making the entire list not match if there’s any null values returned from SELECT id FROM master.

See if there are any rows returned from this query:

SELECT id
FROM master
WHERE id is null;

If not, then this isn’t your problem.

If there are values, then the fix is to exclude null ids from the list:

INSERT INTO master
SELECT *
FROM temp 
WHERE id NOT IN (SELECT id FROM master where id is not null)

The other thing to consider is that there are simply no values not already inserted!