Question

For my own amusement, I’ve cooked up a python script that allows me to use python for bash one-liners; Supply a python generator expression; and the script iterates over it. Here’s the script:

DEFAULT_MODULES = ['os', 're', 'sys']

_g = {}
for m in DEFAULT_MODULES:
    _g[m] = __import__(m)

import sys
sys.stdout.writelines(eval(sys.argv[1], _g))

And here’s how you might use it.

$ groups | python pype.py '(l.upper() for l in sys.stdin)'
DBORNSIDE
$ 

For the intended use, it works perfectly!

But when I don’t feed it with pipe and just invoke it directly, for instance: _{[emphasis added to show what I type]}

$ python pype.py '("%r\n" % (l,) for l in sys.stdin)'
fooEnter
barEnter
bazEnter
Ctrl DCtrl D'foo\n'
'bar\n'
'baz\n'
$ 

In order to stop accepting input and produce any output, I have to type either Enter – Ctrl D – Ctrl D or Ctrl D – Ctrl D – Ctrl D. This violates my expectations, that each line should be processed as entered, and that typing Ctrl D at any time will end the script. Where is the gap in my understanding?

EDIT: I’ve updated the interactive example to show that I’m not seeing the quoting wim describes in his answer, and some more examples too.

$ python pype.py '("%r\n" % (l,) for l in sys.stdin)'
fooCtrl DCtrl DbarEnter
Ctrl DCtrl D'foobar\n'
$ python pype.py '("%r\n" % (l,) for l in sys.stdin)'
fooCtrl VCtrl D^DbarEnter
Ctrl DCtrl D'foo\x04bar\n'
$ 

Answer 1

Ctrl-D is recognized not necessarily as EOF, but as “terminate current read() call”.

If you have an empty line (or just pressed Ctrl-D) and press Ctrl-D, your read() terminates immediately and returns 0 read bytes. And this is a sign for EOF.

If you have data in a line and press Ctrl-D, your read() terminates with whatever there has been typed, of course without a terminating newline ('\n').

So if you have input data, you press Ctrl-D twice of a non-empty line or once on a empty one, i.e. with Enter before.

This all holds for the normal OS interface, accessible from Python via os.read().

Python file objects, and also file iterators, treat the first EOF recognized as termination for the current read() call, as they suppose there is nothing any longer. A next read() call tries again and needs another Ctrl-D in order to really return 0 bytes. The reason is that a file object read() always tries to return as many bytes as requested and tries to fill up if a OS read() returns less than requested.

As opposite to file.readline(), iter(file) uses the internal read() functions to read and thus always has this special requirement of the extra Ctrl-D.

I always use iter(file.readline, '') to read line-wise from a file.