Question

Python: How to remove all duplicate items from a list

Hey guys

I have a list of (file,inode,image,hash)-tuples. I need to delete BOTH items if they have the same hash. I don’t have that much of programming experience, so maybe a hint for what i have to look would already be helpful.
I’ve already searched the Internet, but the only thing i found was this.
So far I’ve come up with this (extremely awkward) solution:

        hashlist = {}
        files_tobe_removed = []
        for (file, inode, image, hash) in self.files_for_json:
            hashlist[hash] = 0
        for (file, inode, image, hash) in self.files_for_json:
            hashlist[hash] +=1
        for (k,v) in hashlist.iteritems():
            if v == 2:
                files_tobe_removed.append(k)
        for (file,inode,image,hash) in self.files_for_json:
            if hash in files_tobe_removed:
                path = self.outDir + file
                os.remove(path)
                self.files_for_json.remove((file,inode,image,hash))

Any help will be appreciated. Thanks in advance

Answer 1

>>> from collections import Counter
>>> L=[1,2,3,4,5,6,7,8,9,1,2,3,4,5,6,1,2,3]
>>> [k for k,v in Counter(L).items() if v==1]
[7, 8, 9]

To clarify:

hash_counter = Counter(x[3] for x in self.files_for_json)
for (file,inode,image,hash) in self.files_for_json:
    if hash_counter[hash]>1:
        # duplicated hash
        ...