Question

I have written basic python snippets to first insert values in a list and then reverse them. I found that there was a huge difference of speed of execution between insert and append methods.

Snippet 1:

L = []
for i in range(10**5):
 L.append(i)
L.reverse()

Time taken to execute this :

real    0m0.070s
user    0m0.064s
sys         0m0.008s

Snippet 2:

l = []
for i in range(10**5):
 l.insert(0,i)

Time taken to execute:

real    0m5.645s
user    0m5.516s
sys         0m0.020s

I expected the snippet 2 to perform much better than snippet1, since I am performing the reverse operation directly by inserting the numbers before. But the time taken says otherwise. I fail to understand why the latter method takes more time to execute, even though the method looks more elegant. Does any one have any explanation for this?

Answer 1

Note that your results will depend on the precise Python implementation. cpython (and pypy) automatically resize your list and overprovision space for future appends and thereby speed up the append furthermore.

Internally, lists are just chunks of memory with a constant size (on the heap). Sometimes you’re lucky and can just increase the size of the chunk, but in many cases, an object will already be there. For example, assume you allocated a chunk of size 4 for a list [a,b,c,d], and some other piece of code allocated a chunk of size 6 for a dictionary:

Memory  0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
       |a b c d| | dictionary |

Assume your list has 4 elements, and another one is added. Now, you can simply resize the list to size 5:

Memory  0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
       |a b c d e| dictionary |

However, what do you do if you need another element now?

Well, the only thing you can do is acquire a new space and copy the contents of the list.

Memory 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
                | dictionary |a  b  c  d  e  f |

Note that if you acquire space in bulk (the aforementioned overprovisioning), you’ll only need to resize (and potentially copy) the list every now and then.

In contrast, when you insert at position 0, you always need to copy your list. Let’s insert x:

Memory  0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
orig   |a b c d| |dictionary|
after  |x a b c d|dictionary|

Although there was enough space to append x at the end, we had to move (not even copy, which may be less expensive in memory) all the other values.