Question

In sum: I need to write a List Comprehension in which i refer to list that is being created by the List Comprehension.

This might not be something you need to do every day, but i don’t think it’s unusual either.

Maybe there’s no answer here–still, please don’t tell me i ought to use a for loop. That might be correct, but it’s not helpful. The reason is the problem domain: this line of code is part of an ETL module, so performance is relevant, and so is the need to avoid creating a temporary container–hence my wish to code this step in a L/C. If a for loop would work for me here, i would just code one.

In any event, i am unable to write this particular list comprehension. The reason: the expression i need to write has this form:

[ some_function(s) for s in raw_data if s not in this_list ]

In that pseudo-code, “this_list” refers to the list created by evaluating that list comprehension. And that’s why i’m stuck–because this_list isn’t built until my list comprehension is evaluated, and because this list isn’t yet built by the time i need to refer to it, i don’t know how to refer to it.

What i have considered so far (and which might be based on one or more false assumptions, though i don’t know exactly where):

• doesn’t the python interpreter have
  to give this list-under-construction
  a name? i think so

• that temporary name is probably taken
  from some bound method used to build
  my list (‘sum’?)

• but even if i went to the trouble to
  find that bound method and assuming
  that it is indeed the temporary name
  used by the python interpreter to
  refer to the list while it is under
  construction, i am pretty sure you
  can’t refer to bound methods
  directly; i’m not aware of such an
  explicit rule, but those methods (at
  least the few that i’ve actually
  looked at) are not valid python
  syntax. I’m guessing one reason why
  is so that we do not write them into
  our code.

so that’s the chain of my so-called reasoning, and which has led me to conclude, or at least guess, that i have coded myself into a corner. Still i thought i ought to verify this with the Community before turning around and going a different direction.

Answer 1

There used to be a way to do this using the undocumented fact that while the list was being built its value was stored in a local variable named _[1].__self__. However that quit working in Python 2.7 (maybe earlier, I wasn’t paying close attention).

You can do what you want in a single list comprehension if you set up an external data structure first. Since all your pseudo code seemed to be doing with this_list was checking it to see if each s was already in it — i.e. a membership test — I’ve changed it into a set named seen as an optimization (checking for membership in a list can be very slow if the list is large). Here’s what I mean:

raw_data = [c for c in 'abcdaebfc']

seen = set()
def some_function(s):
    seen.add(s)
    return s

print [ some_function(s) for s in raw_data if s not in seen ]
# ['a', 'b', 'c', 'd', 'e', 'f']

If you don’t have access to some_function, you could put a call to it in your own wrapper function that added its return value to the seen set before returning it.

Even though it wouldn’t be a list comprehension, I’d encapsulate the whole thing in a function to make reuse easier:

def some_function(s):
    # do something with or to 's'...
    return s

def add_unique(function, data):
    result = []
    seen = set(result) # init to empty set
    for s in data:
        if s not in seen:
            t = function(s)
            result.append(t)
            seen.add(t)
    return result

print add_unique(some_function, raw_data)
# ['a', 'b', 'c', 'd', 'e', 'f']

In either case, I find it odd that the list being built in your pseudo code that you want to reference isn’t comprised of a subset of raw_data values, but rather the result of calling some_function on each of them — i.e. transformed data — which naturally makes one wonder what some_function does such that its return value might match an existing raw_data item’s value.