Question

I have two lists of dictionaries

list1 = [ {..}, {..}, ..]
list2 = [ {..}, {..}, ..]

I want to remove the dictionaries in list1 which are in list2. I had a similar problem where I had a list of lists instead of a dictionary and it is solved here

python function slowing down for no apparent reason

If I use the same code which is,

def removeDups(list1, list2):
    list2_set = set([tuple(x) for x in list2])
    diff = [x for x in list1 if tuple(x) not in list2_set]

    return diff

I do not get correct results since dictionaries like

{key1:'a', key2:'b'} and 
{key2:'b', key1:'a'}

which are the same are actually considered as different. How can I change the code or what can I do to remove dictionaries from list1 that appear in list2?

Answer 1

You can’t use dicts in sets because they’re mutable and don’t have stable identities. You can work around that by making a tuple out of their items. Note that simply wrapping a dict in a tuple doesn’t get around the fact that distinct dicts will still appear to be distinct objects even if they contain the same items.

To turn two “equivalent” dicts into equal objects, take all of their items, sort the items, and then stuff them into a tuple: tuple(sorted(map.items())). Those tuples will properly compare equal to each other if they contain the same items, no matter the order of the original dict.

def removeDups(list1, list2):
    set1 = set(tuple(sorted(x.items())) for x in list1)
    set2 = set(tuple(sorted(x.items())) for x in list2)

    return set1 - set2