Question

My intent is to create a dictionary containing whose keys are primitives and whose values are zero-argument functions which return strings. (This is part of a larger project to implement a VM.) Certain of these functions are non-trivial, and created and assigned manually. Those work fine. Others, however, seem amenable to automatic generation.

My first attempt failed:

>>> regs = ['a', 'b', 'c', 'x', 'y', 'z']
>>> vals = {i : lambda: r for i, r in enumerate(regs)}
>>> [(k, vals[k]()) for k in vals.keys()]
[(0, 'z'), (1, 'z'), (2, 'z'), (3, 'z'), (4, 'z'), (5, 'z')]

OK, fine; the lambda function doesn’t read r until it’s called. I tried again, trying to isolate a value on its own:

>>> from copy import copy
>>> vals = {}
>>> i = 0
>>> for reg in regs:
...     r = copy(reg)  # (1)
...     vals[i] = lambda: r
...     i += 1
...
>>> [(k, vals[k]()) for k in vals.keys()]
[(0, 'z'), (1, 'z'), (2, 'z'), (3, 'z'), (4, 'z'), (5, 'z')]

(1) I thought this step would create an independent variable that wouldn’t change when reg did. That turns out not to be the case.

So that attempt clearly didn’t work. Maybe copy, on a string, is a noop?

>>> 's' is 's'
True
>>> a = 's'
>>> b = copy(a)
>>> a is b
True
>>> from copy import deepcopy
>>> b = deepcopy(a)
>>> a is b
True

Right. Copy, on a string, is a noop. Deepcopy doesn’t fix this. In consequence, the lambda still has a reference to the variable which is being updated on each loop, causing this bug.

We need a different approach. What if I save the variable I want to a static variable of the temporary function? That should work if each temporary function gets its own identity…

>>> vals = {}
>>> i = 0
>>> for reg in regs:
...     def t():
...             return t.r
...     t.r = reg
...     vals[i] = t
...     i += 1
...
>>> [(k, vals[k]()) for k in vals.keys()]
[(0, 'z'), (1, 'z'), (2, 'z'), (3, 'z'), (4, 'z'), (5, 'z')]

Nope. At this point, I’m on the verge of just handling it all manually:

>>> vals = {}
>>> vals[0] = lambda: 'a'
>>> vals[1] = lambda: 'b'

… and so forth. This feels like giving up, though, and will be incredibly tedious. Is there a proper pythonic way to accomplish this task? After all, one of the reasons I usually love python is that I get to stay away from manual pointer management; I never imagined I’d wish it included a full suite of pointer tools!

Answer 1

Closures never copy, they copy neither values nor references. Instead, they remember the scope and variable they use, and always go back there. The same is true in several other languages, such as C#, JavaScript, IIRC Lisp and probably more. The same is true in several other languages. This is important for some advanced use cases (basically every time you want several closures to share state), but it can bite one. For instance:

x = 1
def f(): return x
x = 2
assert f() == 2

As Python only creates new scopes for functions (and classes and modules, but that doesn’t matter here), the loop variable reg exists only once and thus all closures refer to the same variable. Therefore, when called after the loop, they see the last value the variable assumes.

The same is true, only this time it’s the t which is shared – you do create N separate closures, each with the right value in its r attribute. But when called, they look up t in the enclosing scope, and thus always get a reference to the last closure you created.

There are several workarounds. One is pushing the closure creation into a separate function, which forces a new, dedicated scope for each closure to refer to:

def make_const(x):
    def const():
        return x
    return const

Another possibility (terser but more obscure) is (ab-)using the fact that default parameters are bound at definition time:

for reg in regs:
    t = lambda reg=reg: reg

In other cases, you can use functools, but it does not seem to apply here.