Question

I am getting some very surprising results that seem to indicate that it’s more efficient to wrap an iterator in list and get it’s length compared to walking it with a lambda. How is this possible? Intuition would suggest that allocating all these lists would be slower.

And yes – I am aware that you can’t always do this as iterators can be infinite. 🙂

from itertools import groupby
from timeit import Timer

data = "abbbccacabbbccacabbbccacabbbccacabbbccacabbbccacabbbccacabbbccacabbbccacabbbccacabbbccacabbbccacabbbccacabbbccacabbbccacabbbccac" 

def rle_walk(gen):
    ilen = lambda gen : sum(1 for x in gen)
    return [(ch, ilen(ich)) for ch,ich in groupby(data)]

def rle_list(data):
    return [(k, len(list(g))) for k,g in groupby(data)]

# randomy data
t = Timer('rle_walk("abbbccacabbbccacabbbccacabbbccacabbbccacabbbccacabbbccacabbbccacabbbccacabbbccacabbbccacabbbccacabbbccacabbbccacabbbccacabbbccac")', "from __main__ import rle_walk; gc.enable()")
print t.timeit(1000)

t = Timer('rle_list("abbbccacabbbccacabbbccacabbbccacabbbccacabbbccacabbbccacabbbccacabbbccacabbbccacabbbccacabbbccacabbbccacabbbccacabbbccacabbbccac")', "from __main__ import rle_list; gc.enable()")
print t.timeit(1000)

# chunky blocks
t = Timer('rle_walk("aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaabbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbccccccccccccccccccccccccccccccccccccccccccccc")', "from __main__ import rle_walk; gc.enable()")
print t.timeit(1000)

t = Timer('rle_list("aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaabbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbccccccccccccccccccccccccccccccccccccccccccccc")', "from __main__ import rle_list; gc.enable()")
print t.timeit(1000)

1.42423391342
0.145968914032
1.41816806793
0.0165541172028

Answer 1

Unfortunately your rle_walk has a bug; it takes parameter gen but should take parameter data, so it’s operating on the wrong input. Also, it’s unfair to make rle_walk use a lambda where rle_list works inline. Rewriting like so:

def rle_walk(data):
    return [(k, sum(1 for _ in g)) for k, g in groupby(data)]

def rle_list(data):
    return [(k, len(list(g))) for k, g in groupby(data)]

and testing:

data_block = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaabbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbccccccccccccccccccccccccccccccccccccccccccccc"
data_random = "abbbccacabbbccacabbbccacabbbccacabbbccacabbbccacabbbccacabbbccacabbbccacabbbccacabbbccacabbbccacabbbccacabbbccacabbbccacabbbccac"
print [[Timer('r("{data}")'.format(data=data),
              "from __main__ import {r} as r; gc.enable()".format(r=r)).timeit(1000)
        for r in ['rle_walk', 'rle_list']]
        for data in (data_block, data_random)]

gives

[[0.02709507942199707, 0.022060155868530273],
 [0.12022995948791504, 0.16360306739807129]]

so we see that walk is slightly slower than list on the blocky data, but slightly faster on the random data. I’d guess the reason is that generators (in Python) impose an overhead compared to the list constructor; and the memory overhead of a 30-item list is too small to impose any significant penalty.

Disassembling the functions provides a little insight:

>>> dis.dis(lambda g: (1 for _ in g))
  1           0 LOAD_CONST               0 (<code object <genexpr> at 0x2b9202a6fe40, file "<stdin>", line 1>)
              3 MAKE_FUNCTION            0
              6 LOAD_FAST                0 (g)
              9 GET_ITER            
             10 CALL_FUNCTION            1
             13 RETURN_VALUE        
>>> dis.dis((lambda g: (1 for _ in g)).func_code.co_consts[0])
  1           0 SETUP_LOOP              18 (to 21)
              3 LOAD_FAST                0 (.0)
        >>    6 FOR_ITER                11 (to 20)
              9 STORE_FAST               1 (_)
             12 LOAD_CONST               0 (1)
             15 YIELD_VALUE         
             16 POP_TOP             
             17 JUMP_ABSOLUTE            6
        >>   20 POP_BLOCK           
        >>   21 LOAD_CONST               1 (None)
             24 RETURN_VALUE        
>>> dis.dis(lambda g: len(list(g)))
  1           0 LOAD_GLOBAL              0 (len)
              3 LOAD_GLOBAL              1 (list)
              6 LOAD_FAST                0 (g)
              9 CALL_FUNCTION            1
             12 CALL_FUNCTION            1
             15 RETURN_VALUE        

The much larger code volume for the generator form is going to have some effect; while the list form has an O(log n) factor for constructing the throwaway list it’s going to be dominated by the k*O(n) factors in looping the various iterators. One thing to take away from this is that memory allocation is fast, at least for small (sub-page) allocations in a single-threaded environment (which CPython is by necessity of the GIL).