Question

I am looking a method to removes all duplicate points from a a X,Y,Z file. What i wish to code is remove points that have identical x and y coordinates. The first point survives, all subsequent duplicates are removed.

import numpy as np
import matplotlib.pyplot as plt
import matplotlib.mlab as ml
import matplotlib.delaunay
from matplotlib.mlab import griddata

# my area boundary box
xmax, xmin, ymax, ymin = 640000.06, 636999.83, 6070000.3, 6066999.86

# generate fake data
ndata = 500000
# Generate random data to simulate
x = np.random.randint(xmin, xmax, ndata)
y = np.random.randint(ymin, ymax, ndata)
z = np.random.randint(0,20,ndata)
mypoints = zip(x,y,z)

Thanks in advance for helps and tips!!!
🙂

Answer 1

If you run Python 2.7 or higher, you could use an OrderedDict as a filter:

from collections import OrderedDict

tmp = OrderedDict()
for point in zip(x, y, z):
    tmp.setdefault(point[:2], point)

mypoints = tmp.values()

Apart from filtering, this also preserves the order of the random sequences.

---

Another receipie can be found at the Python documentation, which can be translated to something like:

from itertools import groupby

keyfunc = lambda p: p[:2]
mypoints = []
for k, g in groupby(sorted(zip(x, y, z), key=keyfunc), keyfunc):
    mypoints.append(list(g)[0])