Question

I am writing a bit of python code where I had to check if all values in list2 was present in list1, I did that by using set(list2).difference(list1) but that function was too slow with many items in the list.

So I was thinking that list1 could be a dictionary for fast lookup…

So I would like to find a fast way to determent if a list has an item that isn’t part of a dict

performance wise is there any difference between

d = {1: 1, 2:2, 3:3}
l = [3, 4, 5]

for n in l:
  if not n in d:
    do_stuff

vs

for n in l:
  if not d[n]:
    do_stuff

and please if both of these are rubbish and you know something much quicker, tell me.

Edit1: list1 or d can contain elements not in list2 but not the other way around.

Answer 1

A fast way to achieve what you want will be using all and a generator comprehension.

s_list2 = set(list2)
all_present = all(l in s_list2 for l in list1)

This will be advantageous in the case that some elements of list1 are not present in list2.

Some timing. In the case where all values in the first list are contained in the second:

In [4]: l1 = range(100)
In [5]: l2 = range(1000)
In [6]: random.shuffle(l1)
In [9]: random.shuffle(l2)
In [20]: %timeit s2 = set(l2); all(l in s2 for l in l1)
10000 loops, best of 3: 26.4 us per loop
In [21]: %timeit s1 = set(l1); s2 = set(l2); s1.issubset(s2)
10000 loops, best of 3: 25.3 us per loop

If we look at the case where some values in the first list are not present in the second:

In [2]: l1 = range(1000)
In [3]: l2 = range(100)
In [4]: random.shuffle(l1)
In [5]: random.shuffle(l2)
In [6]: sl2 = set(l2)
In [8]: %timeit ss = set(l2); set(l1) & ss == ss
10000 loops, best of 3: 27.8 us per loop
In [10]: %timeit s1 = set(l1); s2 = set(l2); s2.issubset(s1)
10000 loops, best of 3: 24.7 us per loop
In [11]: %timeit sl2 = set(l2); all(l in sl2 for l in l1)
100000 loops, best of 3: 3.58 us per loop

You can see that this method is equivalent in performance to the issubset in the first case and is faster in the second case as it will short circuit and obviates the need to construct 2 intermediate sets (only requiring one).

Having one large list and one small lists demonstrates the benefit of the gencomp method:

In [7]: l1 = range(10)
In [8]: l2 = range(10000)
In [9]: %timeit sl2 = set(l2); all(l in sl2 for l in l1)
1000 loops, best of 3: 230 us per loop
In [10]: %timeit sl1 = set(l1); all(l in sl1 for l in l2)
1000000 loops, best of 3: 1.45 us per loop
In [11]: %timeit s1 = set(l1); s2 = set(l2); s1.issubset(s2)
1000 loops, best of 3: 228 us per loop
In [12]: %timeit s1 = set(l1); s2 = set(l2); s2.issubset(s1)
1000 loops, best of 3: 228 us per loop