Question

biglist = 

[ 

    {'title':'U2 Band','link':'u2.com'}, 
    {'title':'ABC Station','link':'abc.com'}, 
    {'title':'Live Concert by U2','link':'u2.com'} 

]

I would like to remove the THIRD element inside the list…because it has “u2.com” as a duplicate. I don’t want duplicate “link” element. What is the most efficient code to do this so that it results in this:

biglist = 

[ 

    {'title':'U2','link':'u2.com'}, 
    {'title':'ABC','link':'abc.com'}
]

I have tried many ways, including using many nested “for …in ….” but this is very inefficient and too long.

Answer 1

You can sort the list, using the link field of each dictionary as the sort key, then iterate through the list once and remove duplicates (or rather, create a new list with duplicates removed, as is the Python idiom), like so:

# sort the list using the 'link' item as the sort key
biglist.sort(key=lambda elt: elt['link'])

newbiglist = []
for item in biglist:
    if newbiglist == [] or item['link'] != newbiglist[-1]['link']:
        newbiglist.append(item)

This code will give you the first element (relative ordering in the original biglist) for any group of “duplicates”. This is true because the .sort() algorithm used by Python is guaranteed to be a stable sort — it does not change the order of elements determined to be equal to one another (in this case, elements with the same link).