Question

I am fairly new to Python and I am interested in listing duplicates within a list. I know how to remove the duplicates ( set() ) within a list and how to list the duplicates within a list by using collections.Counter; however, for the project that I am working on this wouldn’t be the most efficient method to use since the run time would be n(n-1)/2 –> O(n^2) and n is anywhere from 5k-50k+ string values.

So, my idea is that since python lists are linked data structures and are assigned to the memory when created that I begin counting duplicates from the very beginning of the creation of the lists.

1. List is created and the first index value is the word ‘dog’
2. Second index value is the word ‘cat’
3. Now, it would check if the second index is equal to the first index, if it is then append to another list called Duplicates.
4. Third index value is assigned ‘dog’, and the third index would check if it is equal to ‘cat’ then ‘dog’; since it matches the first index, it is appended to Duplicates.
5. Fourth index is assigned ‘dog’, but it would check the third index only, and not the second and first, because now you can assume that since the third and second are not duplicates that the fourth does not need to check before, and since the third/first are equal, the search stops at the third index.

My project gives me these values and append it to a list, so I would want to implement that above algorithm because I don’t care how many duplicates there are, I just want to know if there are duplicates.

I can’t think of how to write the code, but I figured the basic structure of it, but I might be completely off (using random numgen for easier use):

for x in xrange(0,10):
    list1.append(x)
    for rev, y in enumerate(reversed(list1)):
        while x is not list1(y):
            cond()
            if ???

Answer 1

I really don’t think you’ll get better than a collections.Counter for this:

c = Counter(mylist)
duplicates = [ x for x,y in c.items() if y > 1 ]

building the Counter should be O(n) (unless you’re using keys which are particularly bad for hashing — But in my experience, you need to try pretty hard to make that happen) and then getting the duplicates list is also O(n) giving you a total complexity of O(2n) == O(n) (for typical uses).