Question

EDIT: as @BrenBarn pointed out, the original didn’t make sense.

Given a list of dicts (courtesy of csv.DictReader–they all have str keys and values) it’d be nice to remove duplicates by stuffing them all in a set, but this can’t be done directly since dict isn’t hashable. Some existing questions touch on how to fake __hash__() for sets/dicts but don’t address which way should be preferred.

# i. concise but ugly round trip
filtered = [eval(x) for x in {repr(d) for d in pile_o_dicts}]

# ii. wordy but avoids round trip
filtered = []
keys = set()
for d in pile_o_dicts:
    key = str(d)
    if key not in keys:
        keys.add(key)
        filtered.append(d)

# iii. introducing another class for this seems Java-like?
filtered = {hashable_dict(x) for x in pile_o_dicts}

# iv. something else entirely

In the spirit of the Zen of Python what’s the “obvious way to do it”?

Answer 1

The artfully named pile_o_dicts can be converted to a canonical form by sorting their items lists:

 groups = {}
 for d in pile_o_dicts:
     k = tuple(sorted(d.items()))
     groups.setdefault(k, []).append(d)

This will group identical dictionaries together.

FWIW, the technique of using sorted(d.items()) is currently used in the standard library for functools.lru_cache() in order to recognize function calls that have the same keyword arguments. IOW, this technique is tried and true 🙂