Question

I am developing a GUI to launch an external long-term running background program. This background program can be given input command via stdin and use stdout and stderr to keep printing out output and error messages. I use a wx.TextCtrl object inside the GUI to give input and print output. My current code is as follows, which is mainly inspired by the “how to implemente a shell GUI window” post: wxPython: how to create a bash shell window?

However, my following code uses “buffer previous output” approach, i.e., I use a thread to buffer the output. The buffered transaction output could be only rendered when I give next input command and press “return” button. Now, I want to see the output messages timely, therefore I want to have the feature that “the output can always be spontaneously printed out (directly flushed out) from the background subprocess and I could also interfere to type some input command via stdin and have the output printed.

class BashProcessThread(threading.Thread):
    def __init__(self, readlineFunc):
        threading.Thread.__init__(self)
        self.readlineFunc = readlineFunc
        self.lines = []
        self.outputQueue = Queue.Queue()
        self.setDaemon(True)

    def run(self):
        while True:
           line = self.readlineFunc()
           self.outputQueue.put(line)
           if (line==""):
            break
        return ''.join(self.lines)

    def getOutput(self):
        """ called from other thread """            
        while True:
            try:
                line = self.outputQueue.get_nowait()
                lines.append(line)
            except Queue.Empty:
                break
        return ''.join(self.lines)

class myFrame(wx.Frame):
    def __init__(self, parent, externapp):
        wx.Window.__init__(self, parent, -1, pos=wx.DefaultPosition)
        self.textctrl = wx.TextCtrl(self, style=wx.TE_PROCESS_ENTER|wx.TE_MULTILINE)
        launchcmd=["EXTERNAL_PROGRAM_EXE"]
        p = subprocess.Popen(launchcmd, stdin=subprocess.PIPE, 
                stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        self.outputThread = BashProcessThread(p.stdout.readline)
        self.outputThread.start()
        self.__bind_events()         
        self.Fit()

    def __bind_events(self):
        self.Bind(wx.EVT_TEXT_ENTER, self.__enter)

    def __enter(self, e):
        nl=self.textctrl.GetNumberOfLines()
        ln =  self.textctrl.GetLineText(nl-1)
        ln = ln[len(self.prompt):]     
        self.externapp.sub_process.stdin.write(ln+"\n")
        time.sleep(.3)
        self.textctrl.AppendText(self.outputThread.getOutput())

How should I modify the above code to achieve this? Do I still need to use thread? Can I code a thread as follows?

class PrintThread(threading.Thread):
    def __init__(self, readlineFunc, tc):
        threading.Thread.__init__(self)
        self.readlineFunc = readlineFunc
        self.textctrl=tc
        self.setDaemon(True)

    def run(self):
        while True:
            line = self.readlineFunc()
            self.textctrl.AppendText(line)

However, when I tried with the above code, it crashes.

I have errors from Gtk, as follows.

(python:13688): Gtk-CRITICAL **: gtk_text_layout_real_invalidate: assertion `layout->wrap_loop_count == 0' failed
Segmentation fault

or sometimes the error

 (python:20766): Gtk-CRITICAL **: gtk_text_buffer_get_iter_at_mark: assertion `GTK_IS_TEXT_MARK (mark)' failed
Segmentation fault

or sometimes the error

(python:21257): Gtk-WARNING **: Invalid text buffer iterator: either the iterator is uninitialized, or the characters/pixbufs/widgets in the buffer have been modified since the iterator was created.
You must use marks, character numbers, or line numbers to preserve a position across buffer modifications.
You can apply tags and insert marks without invalidating your iterators,
but any mutation that affects 'indexable' buffer contents (contents that can be referred to by character offset)
will invalidate all outstanding iterators
Segmentation fault

or sometimes the error

Gtk-ERROR **: file gtktextlayout.c: line 1113 (get_style): assertion failed: (layout->one_style_cache == NULL)
aborting...
Aborted

or other error message, but different error message each time, really weird!

It seems the wx.TextCtrl or the underlying gui control of GTK+ has some problem with multi-threading. sometimes I do not type any input commands, it also crashes. I search from the certain post on the internet, and looks like it is dangerous to call a GUI control from a secondary thread.

I found my mistake. As pointed out in wxpython — threads and window events or in chapter 18 of the book “WxPython in action” by Noel and Robin:

The most important point is that GUI operations must take place in the main thread, or the one that the application loop is running in. Running GUI operations in a separate thread is a good way for your application to crash in unpredictable and hard-to-debug ways…

My mistake is I tried to pass the wx.TextCtrl object to another thread. This is not right. I will reconsider my design.

Answer 1

The biggest problem is that you can’t force the subprocess not to buffer its output, and the standard I/O library of most programs will buffer the output when stdout is a pipe (more accurately, they will go from line-buffering to block buffering). Tools like Expect fix this by running the subprocess in a pseudo-tty, which basically tricks the subprocess into thinking its output is going to a terminal.

There is a Python module called Pexpect, which solves this in the same way that Expect does. I’ve never used it, so caveat emptor.