Question

I wrote this little test class, based upon a Python issue – closed/fixed, and it seems to be occurring in Python 2.7.1 on Fedora 15.

from subprocess import Popen, PIPE
from threading import Thread

OUTPUT = "asl;dkfhqwiouethnjzxvnhsldfhuyasdhofuiweqamchuisemfawepfhuaipwemhfuaehfclkuehnflw ehfcoiuwehfiuwmhefuiwehfoiuhSfcl hfulh fuiqhuif huiwafh uiahf iUH fhh flkJH fl HASLFuhSAIULFhSUA HFulSAHfOI SUFChiuwqhncriouweycriuowanbyoUIWCryu iWyruawyrouiWYRcoiu YCRoiuNr uyr oUIAWryocIUWRNyc owuroiuNr cuWyrnawueitcnoy U IuiR yiuowaYnorc oWIUAr coiury iuoAW rnuoi asdfsdfd\n"


class X(Thread):
    def __init__(self):
        Thread.__init__(self)

    def run(self):
        print("Running")
        for i in xrange(10):
            s = Popen(
                "cat /tmp/junk",
                shell=True,
                stdout=PIPE,
                universal_newlines=True
            )
            output = s.communicate()[0]
            if not output == OUTPUT:
                print("Error: %r" % output)


XThreads = set([])

for i in xrange(1000):
    XThreads.add(X())

for x in XThreads:
    x.start()

Just create a file, /tmp/junk in this case, that has the contents of OUTPUT, less the last newline.

Running this, you would expect to see “Running” on every line. However, sometimes it shows ” Running” or “RunningRunning\n\nRunning”.

(Removed citation to actual problem, as this was a false-symptom, thanks to @phihag’s answer).

The actual issue: https://stackoverflow.com/questions/9338409/python-subprocess-popen-corrupts-binary-streams

Answer 1

The behavior you’re seeing has nothing to do with subprocess; I can reproduce it with:

import threading
def run(): print("Running")
for x in [threading.Thread(target=run) for i in range(1000)]:
    x.start()

This is because Python’s print is not thread-safe. To avoid the race condition between printing out text and newline, directly write to stdout, like this:

import threading,sys
def run(): sys.stdout.write("Running\n")
for x in [threading.Thread(target=run) for i in range(1000)]:
    x.start()

This assumes the underlying write calls to stdout are thread-safe. That depends on the platform.