Question

hey guys, beginner here. I have written a program that outputs files to .txt’s and am using another to read them and use them. i have used a list to store these values (len(..) gives me 100 for all files). However, whenever i run this:

for w in range(1,20): # i want files file01-file20 excluding file00
    for x in range(100):
        c=c+1 #counter to keep list position on f=0
        exec "f=open('file%02d.txt','r').readlines()"%w #stores data from file00,file01,file02...
        f00=open('file00.txt','r').readlines() #same as ^ but from file00
        for y in range(100):
            xvp=float(f[c].rstrip('\n')) #the error is on this line; the file are stored in vertical order
            pvp=float(f00[y].rstrip('\n')) #maybe even this one
            #and i do stuff with those values...

I get in line 12,
xvp=float(f[c].rstrip(‘\n’))
IndexError: list index out of range

note: there are 100 numbers stored on separate lines in the .txt’s

please, if there is any way to help you help me, let me know
thanks

Answer 1

You seem to be incrementing c two thousand times (20 times 100 — actually only 1900 times, since range(1,20) will not reach the value 20, as you seem to desire in a comment) — so of course you’re going out of range if you use it to index a list of 100! The whole code is rather a mess and I suggest refactoring it radically, to avoid exec and do things the Python way. Assuming Python 2.6 or better (in 2.5, you need a from __future__ import with_statement at the start of your module):

f00 = open('file00.txt').readlines()
for w in range(1, 21):
    for x in range(100):
        with open('file%02d.txt' % w) as f:
            for line in f:
                xvp = float(line)
                for line00 in f00:
                    rvp = float(line00)
                    do_stuff(xvp, rvp)

I don’t know if this is the logic you want — coupling every line of file00.txt with each line from the 20 other files — but at least this makes it clear which lines are coupled up with which;-). If what you want is to only couple the first line of file00.txt with the first line from each of the others, then second line with second lines, etc, then add import itertools at the start of your module and change the contents of the with into:

            for line00, line in itertools.izip(f00, f):
                rvp = float(line00)
                xvp = float(line)
                do_stuff(xvp, rvp)

and so forth.

Note that I’m reading all of file00.txt in memory once and for all (into the f00 list of lines) because apparently you need to loop on those contents more than once, but that’s not needed for the other files.

An obvious optimization is to convert file00.txt‘s lines to floats only once, replacing the f00 = statement with

with open('file00.txt') as f:
  rvps = [float(line) for line in f]

then use rvps directly instead of repeating the conversion every time on the strings in f00 — for example, in the second version (the one using itertools.izip):

            for rvp, line in itertools.izip(rvps, f):
                xvp = float(line)
                do_stuff(xvp, rvp)

Edit: I see I’ve done a number of tiny enhancements while hardly realizing I was doing so, maybe I’d better explain them;-). No need to pass 'r' when opening a file for reading (can’t hurt, but it’s quite idiomatic to omit it). No need to strip trailing (or for that matter leading) whitespace from a string before calling float on it — float happily skips all such leading and trailing whitespace itself. I did fix what apparently was another bug (you’d never deal with file20.txt) by fixing the applicable range to range(1, 21).

The with open(...) as f: statements do the opening, bind name f to the open file object, and, as soon as the block of statements they control is finished, guarantee that the file is properly closed — it should almost invariably be used in preference to a stand-alone open, because ensuring all files are closed ASAP is really very good practice (the with statement has many other excellent use cases, but this is the single most frequent one, and the only one that happens to be necessary for this functionality).

Looping directly on an open file object f (provided the file is opened in text mode, as is the default and applies throughout here), for line in f:, provides one after the other the lines of f (without ever needing to keep them all in memory at once) and is an extremely popular and good Pythonic idiom.

The construct rvps = [float(line) for line in f], which I use in my recommended optimization, is known as a “list comprehension” and it’s a nicely speedy and compact alternative to a loop that builds a new list.

itertools.izip, given a number of iterables, provides a single iterable whose items are tuples made by the items of the other iterables “walked in lockstep”. The built-in zip is similar, but (in Python 2) it builds a list in memory, which itertools.izip avoids, so it’s good practice to learn to use the itertools version to avoid wasting memory (not really important for small files like the ones you have, but good habits are best learned and “just applied” rather than having to reflect on them every single time — just one one doesn’t start every morning pondering whether one should brush one’s teeth, but just goes and does so as a matter of good habit;-).

I’m sure there’s more, but this is what comes to mind off-hand – feel free to ask if I can be of further assistance!