Question

For example, files, in Python, are iterable – they iterate over the lines in the file. I want to count the number of lines.

One quick way is to do this:

lines = len(list(open(fname))) 

However, this loads the whole file into memory (at once). This rather defeats the purpose of an iterator (which only needs to keep the current line in memory).

This doesn’t work:

lines = len(line for line in open(fname)) 

as generators don’t have a length.

Is there any way to do this short of defining a count function?

def count(i):     c = 0     for el in i: c += 1     return c 

To clarify, I understand that the whole file will have to be read! I just don’t want it in memory all at once

Answer 1

Short of iterating through the iterable and counting the number of iterations, no. That’s what makes it an iterable and not a list. This isn’t really even a python-specific problem. Look at the classic linked-list data structure. Finding the length is an O(n) operation that involves iterating the whole list to find the number of elements.

As mcrute mentioned above, you can probably reduce your function to:

def count_iterable(i):     return sum(1 for e in i) 

Of course, if you’re defining your own iterable object you can always implement __len__ yourself and keep an element count somewhere.