Question

In the python program I’m writing, I’ve got a thread which iterates over a large structure in memory and writes it incrementally into a file-like object. I’ve got another thread which takes a file-like object and writes it to disk. Is there an easy way to connect the two, such that any data input from the first thread will be buffered for the second?

Specifically, I’m trying to pass data to subprocess.Popen(). The process will read from stdin, but you cannot pass a ‘file-like’ object to Popen because it calls stdin.fileno() and blows up unless you have a real file.

Instead, you need to pass the PIPE argument to Popen, which allows you to use proc.stdin as a file-like object. But if you’ve already got a file-like object, there doesn’t seem to be a great way to yolk the two of them together.

Answer 1

Use shutil‘s copyfileobj() function:

import shutil import subprocess  proc = subprocess.Popen([...], stdin=subprocess.PIPE)  my_input = get_filelike_object('from a place not given in the question')  shutil.copyfileobj(my_input, proc.stdin) 

No need to use threads.