I would like to be able to spawn a process in python and have two way communication. Of course, Pexpect does this and is indeed a way I might go. However, it is not quite ideal.
My ideal situation would be to have a cross platform generic technique that involved only the standard python libraries. Subprocess gets pretty close, but the fact that I have to wait for the process to terminate before safely interacting with it is not desirable.
Looking at the documentation, it does say there is a stdin,stdout and stderr file descriptors that I can directly manipulate, but there is a big fat warning that says “Don’t Do This”. Unfortunately its not entirely clear why this warning exists, but from what I gather from google is that it is related to os buffering, and it is possible to write code that unexpectedly deadlocks when those internal buffers fail (as a side note, any examples that show the wrong way and right way would be appreciated).
So, risking my code to potential deadlocks, I thought it might be interesting to use poll or select to interactively read from the running process without killing it. Although I lose (i think) the cross platform ability, I like the fact that it requires no additional libraries. But more importantly, I would like to know if this is this a good idea. I have yet to try this approach, but I am concerned about gotchas that could potentially devastate my program. Can it work? What should I test for?
In my specific case I am not really concerned about being able to write to the process, just repeatedly reading from it. Also, I don’t expect my processes to dump huge amounts of text, so I hope to avoid the deadlocking issue, however I would like to know exactly what those limits are and be able to write some tests to see where it breaks down.
Forgive my ignorance on this topic, but couldn’t you just launch python with the -u flag for “unbuffered”?
This might also be of interest…
http://www.gossamer-threads.com/lists/python/python/658167