Question

Possible Duplicate:
List of dictionaries, eliminating duplicates of one key, sorting by another

Does anyone have any clever idea how to remove duplicates from my list of dictionary objects, where two keys should compare to equals and a third is a timestamp? The object that is to be left in the list should be the one with the latest timestamp.

To remove any ambiguity from my question I present a simple test case for my goal:

from datetime import datetime
now = datetime.now()
future = datetime(now.year + 100, *now.timetuple()[1:-2])

# Elements are considered equal if A and B matches.
data = [{"A":10,"B":20,"D":now}, \
        {"A":45,"B":20,"D":now}, \
        {"A":45,"B":20,"D":future}, \
        {"A":66,"B":6,"D":future}, \
        {"A":66,"B":6,"D":now}]

def make_unique(lst):
    pass

make_unique(data)

# data should now contain: (10,20,now),(45,20,future),(66,6,future)
print(data)

Performance is not terrible important, but the list could very well contain a couple of thousends of elements.

Answer 1

You could try to create a dictionary unique and iterate on data. Make the tuple (d['A'], d['B']) your key and the corresponding item d your value. Then, it’s a matter of checking whether (1) the key is already in the dictionary and (2) if the corresponding time stamp is more recent than the one stored:

unique = {}
for d in data:
    key = (d['A'],d['B'])
    if key in unique:
        if unique[key]['D'] < d['D']:
            unique[key] = d
    else:
        unique[key] = d

Then, just retrieve an unsorted list as unique.values().