Question

I’m trying to read a huge amount of lines from standard input with python.

more hugefile.txt | python readstdin.py

The problem is that the program freezes as soon as i’ve read just a single line.

print sys.stdin.read(8)
exit(1)

This prints the first 8 bytes but then i expect it to terminate but it never does. I think it’s not really just reading the first bytes but trying to read the whole file into memory.

Same problem with sys.stdin.readline()

What i really want to do is of course to read all the lines but with a buffer so i don’t run out of memory.

I’m using python 2.6

Answer 1

This should work efficiently in a modern Python:

import sys

for line in sys.stdin:
    # do something...
    print line,

You can then run the script like this:

python readstdin.py < hugefile.txt