Question

I have a script I’m writing to batch download images from an imageboard via JSON/XML api. Previously, it had been purely CLI, but recently I’ve been trying to build a UI in PyQt, with great success but for one problem: thread blocking issues, nonresponsive GUI when actually calling the worker threads within my script. So, I’m trying to switch from threading.Thread to QThread, to make it easier to manage (by emitting a threadFinished SIGNAL to update my GUI), but I can’t seem to get it set up properly. Whenever I run the script, The threads die prematurely. I am running on windows, with PyQt4 on Python 2.7.2.

After a bit more research, I believe the problem lies in a thread exiting, and creating a new thread with a new tuple being passed from the queue. All results I can find online point to it being about the application not exiting cleanly.

Exception KeyError: KeyError(1188,) in <module 'threading' from 'C:\Python27\lib\threading.pyc'> ignored
QObject::killTimers: timers cannot be stopped from another thread

this is the output I receive.

Direct code in question:

md5_queue is queue of an empty dict of md5sum/filename to be filled by Url_Download()
queue is a queue tuple of filenames/urls

in crawler.py:

num_conn = int(max_threads)
threads = []
for download in range(num_conn):
    t = Url_Download(queue, md5_queue)
    t.start()
    threads.append(t)

from functions.py (poorly named, I know) the Url_Download() class:

class Url_Download(QThread):

    file_finished = pyqtSignal(QString, int, name="fileFinished")

    def __init__(self, dl_queue, md5_queue):
        self.dl_queue = dl_queue
        self.md5_queue = md5_queue
        QThread.__init__(self)
    def run(self):
        while 1:
            try:
                count = 0
                file_url, file_path, md5 = self.dl_queue.get_nowait()
                file_extension = str(file_url)[-4:]
                file_name = md5 + file_extension
                while count < 3:
                    count +=1
                    fetch_url(file_url, file_path, md5)
                    if md5 == hash_sum(file_path):
                       self.md5_queue.put_nowait((md5, file_name))
                       self.file_finished.emit("Test!", 10)
                       break

                if count > 3:
                    print 'File failed to download, {} might be corrupt.'.format(file_name)       
                qsize = self.dl_queue.qsize()
                if qsize > 0:
                    print 'Count Remaining: ', qsize
            except Queue.Empty:
                raise SystemExit 
            except:
                traceback.print_exc(file=sys.stderr)
                sys.stderr.flush()

and from GUI.py, the slot connect:

self.connect(self, SIGNAL("fileFinished(QString, int)"), self.handle_test, Qt.QueuedConnection)

Git (testing branch) for the code: https://github.com/CirnoIsTheStrongest/BiriCrawler/tree/testing

Please note, this is my first ever attempt at coding anything. If that is a problem, just let me know

Answer 1

Thanks Avaris for your help, I fixed the connect to point towards my Url_Download() instance. The issue occurring apparently was very inadequately displayed on windows. On my linux VM I was given this error instead:

QThread: Destroyed while thread is still running
QThread: Destroyed while thread is still running
QThread: Destroyed while thread is still running
QThread: Destroyed while thread is still running
Segmentation fault

So the issue was still caused (I believe) by my GUI not waiting for threads to finish their tasks before they were terminated. After referencing my threads object in GUI.py, the error no longer occurs. I’m also finally able to send a signal to the GUI from within my threaded class. Full code changes can be found here for those who want to see the other changes involved: Github page, testing branch

In Crawler.py

---

    threads = []
    for download in range(self.num_of_threads):
        t = Url_Download(self.dl_queue, self.md5_queue, is_cli=True)
        t.start()
        threads.append(t)

    for thread in threads:
        thread.wait()

---

in GUI.py

---

   main = Crawler(gui_tags, gui_limit, gui_page, gui_booru, gui_savepath, gui_partype, gui_rating, max_threads)
    self.threads = main.start_threads()
    for thread in self.threads:
        self.connect(thread, SIGNAL("fileFinished(QString, int)"), self.onFileFinished, Qt.QueuedConnection)
        self.connect(thread, SIGNAL("allFinished()"), self.onAllFilesFinished, Qt.QueuedConnection)

---

in functions.py

---

class Url_Download(QThread):

    file_finished = pyqtSignal(QString, int, name="fileFinished")        

    def __init__(self, dl_queue, md5_queue, is_cli=False, parent=None):
        QThread.__init__(self, parent)
        self.exiting = False
        self.dl_queue = dl_queue
        self.md5_queue = md5_queue
        self.is_cli = is_cli

    def __del__(self):
        self.exiting = True

     def run(self):
        while not self.exiting:
            try:
                count = 0
                file_url, file_path, md5 = self.dl_queue.get_nowait()
                file_extension = str(file_url)[-4:]
                file_name = md5 + file_extension
                while count < 3:
                    count +=1
                    fetch_url(file_url, file_path, md5)
                    if md5 == hash_sum(file_path):
                       self.md5_queue.put_nowait((md5, file_name))
                       self.file_finished.emit("Test!", 10)
                       break

                if self.is_cli:       
                    if count >= 3:
                        print 'File failed to download, {} might be corrupt.'.format(file_name)       
                    qsize = self.dl_queue.qsize()
                    if qsize > 0:
                        print 'Count Remaining: ', qsize
            except Queue.Empty:
                self.__del__()
            except:
                traceback.print_exc(file=sys.stderr)
                sys.stderr.flush()