Question

I am running a Tkinter GUI that spins off another process (python script) with subprocess.Popen(...) and uses pipes for stdout and stderr. Then I’m spinning off a separate thread to asynchronously read the out/err from that process and draw it into a Tkinter Text widget using threading.Thread.

Everything works great except that the async. read thread only executes when I’m moving the mouse or pressing keys on the keyboard. I even put print statements into the threaded function and they start/stop printing when I move the mouse around in circles.

Here’s the async read class that I’m using, borrowed from here:

class AsynchronousFileReader(threading.Thread):
    '''
    Helper class to implement asynchronous reading of a file
    in a separate thread. Pushes read lines on a queue to
    be consumed in another thread.
    '''

    def __init__(self, fd, queue):
        assert isinstance(queue, Queue.Queue)
        assert callable(fd.readline)
        threading.Thread.__init__(self)
        self._fd = fd
        self._queue = queue

    def run(self):
        '''The body of the tread: read lines and put them on the queue.'''
        for line in iter(self._fd.readline, ''):
            self._queue.put(line)

    def eof(self):
        '''Check whether there is no more content to expect.'''
        return not self.is_alive() and self._queue.empty()

And my consume method for pulling messages out of the async file reader (this is the one that runs on a separate thread:

def consume(self, process, console_frame):
    # Launch the asynchronous readers of the process' stdout and stderr.

    stdout_queue = Queue.Queue()
    stdout_reader = AsynchronousFileReader(process.stdout, stdout_queue)
    stdout_reader.start()
    stderr_queue = Queue.Queue()
    stderr_reader = AsynchronousFileReader(process.stderr, stderr_queue)
    stderr_reader.start()

    # Check the queues if we received some output (until there is nothing more to get).
    while not stdout_reader.eof() or not stderr_reader.eof():
        # Show what we received from standard output.
        while not stdout_queue.empty():
            line = stdout_queue.get()
            console_frame.writeToLog(line.strip(), max_lines=None)
            time.sleep(.03) # prevents it from printing out in large blocks at a time

        # Show what we received from standard error.
        while not stderr_queue.empty():
            line = stderr_queue.get()
            console_frame.writeToLog(line.strip(), max_lines=None)
            time.sleep(.03) # prevents it from printing out in large blocks at a time

        # Sleep a bit before asking the readers again.
        time.sleep(.05)

    # Let's be tidy and join the threads we've started.
    stdout_reader.join()
    stderr_reader.join()

    # Close subprocess' file descriptors.
    process.stdout.close()
    process.stderr.close()

    print "finished executing"
    if self.stop_callback:
        self.stop_callback()

Like I said before — the consume() thread only executes when I move the mouse or type on the keyboard — which means the writeToLog(...) function (for appending text into the Tkinter GUI) only gets executed when mouse/keyboard activity happens… Any ideas?

EDIT: I think I might have an idea of what’s happening… If I comment the writeToLog(...) call and replace it with a simple print (taking Tkinter out of the equation) then the consume thread executes normally. It seems Tkinter is the problem here. Any ideas on I can accomplish the Tkinter text-widget update from the consume thread?

EDIT2: Got it working thanks to the comments. Here’s is the final code that I used:

gui_text_queue = Queue.Queue()


def consume(self, process, console_frame):
    # Launch the asynchronous readers of the process' stdout and stderr.

    stdout_queue = Queue.Queue()
    stdout_reader = AsynchronousFileReader(process.stdout, stdout_queue)
    stdout_reader.start()
    stderr_queue = Queue.Queue()
    stderr_reader = AsynchronousFileReader(process.stderr, stderr_queue)
    stderr_reader.start()

    # Check the queues if we received some output (until there is nothing more to get).
    while not stdout_reader.eof() or not stderr_reader.eof():
        # Show what we received from standard output.
        while not stdout_queue.empty():
            line = stdout_queue.get()
            gui_text_queue.put(line.strip())

        # Show what we received from standard error.
        while not stderr_queue.empty():
            line = stderr_queue.get()
            gui_text_queue.put(line.strip())

        # Sleep a bit before asking the readers again.
        time.sleep(.01)

    # Let's be tidy and join the threads we've started.
    stdout_reader.join()
    stderr_reader.join()

    # Close subprocess' file descriptors.
    process.stdout.close()
    process.stderr.close()

    if self.stop_callback:
        self.stop_callback()

Added this method to my Tkinter console frame and called it once at the end of the frame initializer:

def pull_text_and_update_gui(self):
    while not gui_text_queue.empty():
        text = gui_text_queue.get()
        self.writeToLog(text, max_lines=None)
    self.after(5, self.pull_text_and_update_gui)

Answer 1

Tkinter isn’t thread safe. If your writeToLog function tries to insert data into the text widget, you’ll get unpredictable behavior. In order for a separate thread to send data to a widget you’ll need to write the data to a thread-safe queue, then have your main thread poll that queue (using tkinter’s after method).