Question

I’m using ftplib for connecting and getting file list from FTP server.
The problem I have is that the connection hangs from time to time and I don’t know why. I’m running python script as a daemon, using threads.
See what I mean:

def main():
signal.signal(signal.SIGINT, signal_handler)

    app.db = MySQLWrapper()
    try:
        app.opener = FTP_Opener()
        mainloop = MainLoop()

        while not app.terminate:
            # suspend main thread until the queue terminates
            # this lets to restart the queue automatically in case of unexpected shutdown
            mainloop.join(10)
            while (not app.terminate) and (not mainloop.isAlive()):
                time.sleep(script_timeout)
                print time.ctime(), "main: trying to restart the queue"
                try:
                    mainloop = MainLoop()
                except Exception:
                    time.sleep(60)

    finally:
        app.db.close()
        app.db = None
        app.opener = None
        mainloop = None
        try:
            os.unlink(PIDFILE)
        except:
            pass
        # give other threads time to terminate
        time.sleep(1)
        print time.ctime(), "main: main thread terminated"

MainLoop() has some functions for FTP connect, download specific files and disconnect from the server.

Here’s how I get the file’s list:

file_list = app.opener.load_list()

And how FTP_Opener.load_list() function looks like:

def load_list(self):
    attempts = 0
    while attempts<=ftp_config.load_max_attempts:
        attempts += 1
        filelist = []
        try:
            self._connect()
            self._chdir()
            # retrieve file list to 'filelist' var
            self.FTP.retrlines('LIST', lambda s: filelist.append(s))

            filelist = self._filter_filelist(self._parse_filelist(filelist))

            return filelist
        except Exception:
            print sys.exc_info()
            self._disconnect()
            sleep(0.1)

    print time.ctime(), "FTP Opener: can't load file list"
    return []

Why sometimes the FTP connection hangs and how can I monitor this? So if it happens I would like to terminate the thread somehow and start a new one.

Thanks

Answer 1

If you are building for robustness, I would highly recommend that you look into using an event-driven method. One such which have FTP support is Twisted (API).

The advantage is that you don’t block the thread while waiting for i/O and you can create simple timer functions to monitor your connections if you so prefer. It also scales a lot better. It is slightly more complicated to code using event-driven patterns, so if this is just a simple script it may or may not be worth the extra effort, but since you write that you are writing a daemon, it might be worth looking into.

Here is an example of an FTP client: ftpclient.py