Question

I was looking at a bit of code I downloaded from the internet. It’s for a basic webcrawler. I came across the following for loop:

for link in (links.pop(0) for _ in xrange(len(links))):
    ...

Now, I feel the following code will also work:

for link in links:
    ....
links=[]

Researching, I found out that the first instance clears links and also generates a generator object (genexpr). links is never used in the for loop, so its decreasing length has nothing to do with the code.

Is there any particular reason for using the xrange, and popping the elements each time? I.e. Is there any advantage to using a generator object over calling elements of the standard list? Additionally, in what cases would a generator be useful; why?

Answer 1

It’s hard to see any justification for the code you quoted.

The only thing I can think of is that the objects in links might be large, or otherwise associated with scarce resources, and so it might be important to free them as soon as possible (rather than waiting until the end of the loop to free them all). But (a) if so, it would be better to process each link as you created it (perhaps using a generator to organize the code), instead of building up the whole list of links before starting to process it; and (b) even if you had no choice but to build up the whole list before processing it, it would be cheaper to clear each list entry than to pop the list:

for i, link in enumerate(links):
    links[i] = None
    ...

(Popping the first element off a list with n items takes O(n), although in practice it will be fairly fast since it’s implemented using memmove.)

Even if you absolutely insisted on repeatedly popping a list as you iterated across it, it would be better to write the loop like this:

while links:
    link = links.pop(0)
    ...