Question

I usually read files like this in Python:

f = open('filename.txt', 'r')
for x in f:
    doStuff(x)
f.close()

However, this splits the file by newlines. I now have a file which has all of its info in one line (45,000 strings separated by commas). While a file of this size is trivial to read in using something like

f = open('filename.txt', 'r')
doStuff(f.read())
f.close()

I am curious if for a much larger file which is all in one line it would be possible to achieve a similar iteration effect as in the first code snippet but with splitting by comma instead of newline, or by any other character?

Answer 1

The following function is a fairly straightforward way to do what you want:

def file_split(f, delim=',', bufsize=1024):
    prev = ''
    while True:
        s = f.read(bufsize)
        if not s:
            break
        split = s.split(delim)
        if len(split) > 1:
            yield prev + split[0]
            prev = split[-1]
            for x in split[1:-1]:
                yield x
        else:
            prev += s
    if prev:
        yield prev

You would use it like this:

for item in file_split(open('filename.txt')):
    doStuff(item)

This should be faster than the solution that EMS linked, and will save a lot of memory over reading the entire file at once for large files.