Question

How can I make a Python equivalent of pdtolist from Pop-11?

Assume I have a generator called g that returns (say) integers one at a time. I’d like to construct a list a that grows automatically as I ask for values beyond the current end of the list. For example:

print a # => [ 0, 1, 2, g]
print a[0] # => 0
print a[1] # => 1
print a[2] # => 2
# (obvious enough up to here)

print a[6] # => 6
print a # => [ 0, 1, 2, 3, 4, 5, 6, g]
# list has automatically expanded

a = a[4:] # discard some previous values
print a # => [ 4, 5, 6, g]
print a[0] # => 4

Terminology – to anticipate a likely misunderstanding: a list is a “dynamic array” but that’s not what I mean; I’d like a “dynamic list” in a more abstract sense.

To explain the motivation better, suppose you have 999999999 items to process. Trying to fit all those into memory (in a normal list) all at once would be a challenge. A generator solves that part of the problem by presenting them one at a time; each one created on demand or read individually from disk. But suppose during processing you want to refer to some recent values, not just the current one? You could remember the last (say) ten values in a separate list. But a dynamic list is better, as it remembers them automatically.

Answer 1

Many thanks to all who contributed ideas! Here’s what I have gathered together from all the responses. This retains most functionality from the normal list class, adding additional behaviours where necessary to meet additional requirements.

class DynamicList(list):
    def __init__(self, gen):
        self.gen = gen

    def __getitem__(self, index):
        while index >= len(self):
            self.append(next(self.gen))
        return super(DynamicList, self).__getitem__(index)

    def __getslice__(self, start, stop):
        # treat request for "last" item as "most recently fetched"
        if stop == 2147483647: stop = len(self)
        while stop >  len(self):
            self.append(next(self.gen))
        return super(DynamicList, self).__getslice__(start, stop)

    def __iter__(self):
        return self

    def next(self):
        n = next(self.gen)
        self.append(n)
        return n

a = DynamicList(iter(xrange(10)))

Previously generated values can be accessed individually as items or slices. The recorded history expands as necessary if the requested item(s) are beyond the current end of the list. The entire recorded history can be accessed all at once, using print a, or assigned to a normal list using b = a[:]. A slice of the recorded history can be deleted using del a[0:4]. You can iterate over the whole list using for, deleting as you go, or whenever it suits. Should you reach the end of the generated values, StopIteration is raised.

Some awkwardness remains. Assignments like a = a[0:4] successfully truncate the history, but the resulting list no longer auto-expands. Instead use del a[0:4] to retain the automatic growth properties. Also, I’m not completely happy with having to recognise a magic value, 2147483647, representing the most recent item.