Question

I’m in need of a method to quickly return the number of differences between two large lists. The contents of each list item is either 1 or 0 (single integers), and the amount of items in each list will always be 307200.

This is a sample of my current code:

list1 = <list1> # should be a list of integers containing 1's or 0's
list2 = <list2> # same rule as above, in a slightly different order

diffCount = 0
for index, item in enumerate(list1):
    if item != list2[index]:
        diffCount += 1

percent = float(diffCount) / float(307200)

The above works but it is way too slow for my purposes. What I would like to know is if there is a quicker way to obtain the number of differences between lists, or the percentage of items that differ?

I have looked at a few similar threads on this site but they all seem to work slightly different from what I want, and the set() examples don’t seem to work for my purposes. 😛

Answer 1

You can get at least another 10X speedup if you use NumPy arrays instead of lists.

import random
import time
import numpy as np
list1 = [random.choice((0,1)) for x in xrange(307200)]
list2 = [random.choice((0,1)) for x in xrange(307200)]
a1 = np.array(list1)
a2 = np.array(list2)

def foo1():
    start = time.clock()
    counter = 0
    for i in xrange(307200):
        if list1[i] != list2[i]:
            counter += 1
    print "%d, %f" % (counter, time.clock()-start)

def foo2():
    start = time.clock()
    ct = np.sum(a1!=a2)
    print "%d, %f" % (ct, time.clock()-start)
    
foo1() #153490, 0.096215
foo2() #153490, 0.010224