Question

I’m writing a python script that does various permutations of characters. Eventually, the script will crash with out of memory error depending on how much depth I want to go for the permutation.

I had initially thought the solution would have been emptying out the list and restarting over but doing it this way I get index out of bounds error.

This is my current set up:

for j in range(0, csetlen):
    getJ = None
    for i in range(0, char_set_len):
        getJ = word_list[j] + char_set[i]
        word_list.append(getJ)

    csetlen = csetlen - j
    del word_list[j-1:]
    word_list.append(getJ)
    j=0

Basically, csetlen can be a very large number (excess of 100,000,000). Of course I do not have enough RAM for this; so I’m trying to find out how to shrink the list in the outer for loop. How does one do this gracefully?

The memory error has to do with word_list. Currently, I am storing millions of different permutations; I need to be able to “recycle” some of the old list values. How does one do this to a python list?

Answer 1

What you want is an iterator that generates the values on demand (and doesn’t store them in memory):

from itertools import product
getJ_iterator = product(wordlist[:csetlen], char_set[:char_set_len]) 

This is equivalent to the following generator function:

def getJ_gen(first_list, second_list):
    for i in first_list:
        for j in second_list:
            yield (i, j)

getJ_iterator = getJ_gen(wordlist[:csetlen], char_set[:char_set_len])

You would iterate over the object like so:

for item in getJ_iterator:
    #do stuff

Note that item in this case would be a tuple of the form (word, char).