I have a python process running “in the background”. It’s a subprocess.Popen() call started in a totally different process. As I’ve implemented it so far, the two processes are using multiprocessing.connections.(Listener|Client) to communicate with eachother. But I’m open to any way to accomplish the following.
What I want to do is have the “listener” background process use std(in|out|err) as if they’ve come from the “client” process. Streaming is important so that io isn’t read into memory but is read and written efficiently but also to avoid blocking on the listener process. I guess that means I need a lower-level access or some sort of async/threading/multiprocessing solution. That’s what I’m unclear on how do to. For example:
while True: # TODO
conn = listener.accept()
try:
callable_, args, kw = conn.recv()
callable_(*args, **kw)
The question is how to allow callable_ to execute streaming to/from std(in|out|err) efficiently and without blocking. The listener is also long-running, and I’m trying to support arbitrary python code that does from sys import std(in|out|err) at listener module import time will still see std(in|out|err) as if from the client. I can do setup (modifying/replacing sys.std(in|out|err)?) at listener startup before anything else is impored.
Is there a good way to stream this io transparently such that after initial setup the code being run by the listener process can be arbitrary and can treat std(in|out|err) normally without blocking, etc.? Preferably without creating new processes since the goal is to reduce process overhead, though using multiple processes on the client end is preferable since the client is thin. Similarly, if I have to loop over reads/writes of segments of bytes from std(in|out|err) to accomplish effective streaming it would be best to do it in the client.
I’m not entirely sure I understand the question, but have you looked into using select with pipes/sockets to avoid blocking? I’m a bit vague on the other req that it “not be read into memory.”