Question

I have a table set up as follows

id
origin
destination
carrier_id

so typical row could be,

100: London    Manchester  366

Now each route goes both ways, so there shouldn’t be a row like this

233: Manchester    London    366

since that’s essentially the same route (for my purposes anyway)

Unfortunately though, i have wound up with a handful of duplicates. I have over 50,000 routes made up of around 2000 point of origin (or destination, however you want to look at it) in the table. So i’m thinking looping through each point of origin to find duplicates would be insane.

So I don’t even know where to start trying to figure out a query to identify them. Any ideas?

Answer 1

I think you just need a double join, the following will identify all the “duplicate” records joined together.

Here’s an example.

Say SELECT * FROM FLIGHTS yielded:

id  origin   destination  carrierid
1   toronto  quebec      1
2   quebec   toronto     2
3   edmonton calgary     3
4   calgary  edmonton    4
5   hull     vancouver   5
6   vancouveredmonton    6
7   edmonton toronto     7
9   edmonton quebec      8
10   toronto  edmonton  9
11   quebec   edmonton  10
12   calgary  lethbridge 11

So there’s a bunch of duplicates (4 of the routes are duplicates of some other route).

select  *
from    flights t1 inner join flights t2 on t1.origin = t2.destination 
        AND t2.origin = t1.destination

would yield just the duplicates:

id  origin   destination carrierid  id  origin  destination carrierid
1   toronto quebec       1  2   quebec  toronto 2
2   quebec  toronto      2  1   toronto quebec  1
3   edmonton    calgary 3   4   calgary edmonton    4
4   calgary edmonton    4   3   edmonton    calgary 3
7   edmonton    toronto 7   10  toronto edmonton    9
9   edmonton    quebec  8   11  quebec  edmonton    10
10  toronto edmonton    9   7   edmonton    toronto 7
11  quebec  edmonton    10  9   edmonton    quebec  8

At that point you just might delete all the ones that occurred 1st.

delete from flights
where id in (
    select  t1.id
    from    flights t1 inner join flights t2 on t1.origin = t2.destination 
            AND t2.origin = t1.destination
)

Good luck!