Question

It’s the same as finder on Windows but use thread to get faster,

    import os,threading,multiprocessing


    def finder(path,q):
     for x in os.walk(unicode(path)):
      if x[1]:
       for dirname in x[1]:
        if target in dirname.lower():
         q.put(os.path.join(x[0],dirname))
      if x[2]:
       for name in x[2]:
        if target in name.lower():
         q.put(os.path.join(x[0],name))

     q.put(1)

    def printer(q):
     cmd=0
     while 1:
      tmp=q.get()
      if tmp==1:
       cmd += 1
       continue
      if cmd ==thnum:
       break
      print tmp

    if __name__ =="__main__":
     q=multiprocessing.JoinableQueue()
     ini=os.walk(u"C:\\").next()
     thnum=len(ini[1])
     target=raw_input("what you wanna get\n")

     p=multiprocessing.Process(target=printer,args=(q,))
     p.daemon=1
     p.start()

     for i in xrange(thnum):
      t=threading.Thread(target=finder,args=(ini[1][i],q,))
      t.start()
      print i," started"
     q.join()

it shows

0 started
1 started
….
22 started

but never shows the result
so my question is

1. why doesn’t the result shows
2. I know the code is dirty:(…is that a clean way to do it?

thank you guys.

Answer 1

You have just a ton of messy code in here and also some errors. The major problem I see is that your threads are immediately failing to produce anything from their os.walk, and going right to exiting with the q.put. This is because you don’t pass a full path to each thread. Only a directory name. But its hard to know this because you dont use descriptive names for any variables.

Here is a cleaned up version:

import os
import threading
import multiprocessing


def finder(path, q, done):
    for root, dirs, files in os.walk(unicode(path)):
        for dirname in dirs:
            if target in dirname.lower():
                q.put(os.path.join(root,dirname))
        for name in files:
            if target in name.lower():
                q.put(os.path.join(root,name))

    # print "Leaving thread", threading.current_thread()
    done.put(1)

def printer(q,done,worker_count):
    total = 0
    while 1:
        try: done.get_nowait()
        except: pass
        else: total += 1

        if total == worker_count:
            break

        try: tmp=q.get(timeout=1)
        except: pass

        print tmp

if __name__ =="__main__":

    results = multiprocessing.Queue()
    done = multiprocessing.Queue()
    root, dirs, files = os.walk(u"C:\\").next()
    thnum=len(dirs)
    target=raw_input("what you wanna get\n")

    p=multiprocessing.Process(target=printer,args=(results,done,thnum))
    p.start()

    for i in xrange(thnum):
        full_path = os.path.join(root, dirs[i])
        t=threading.Thread(target=finder,args=(full_path, results, done))
        t.start()

    p.join()

See how I join the full path together in the main block before sending them off to each thread? I removed the JoinableQueue because it was never going to do what you think it was. If at any time the printer has cleared out the results queue, but the threads are still trying to find more, the queue will think its done and exit. What I replaced it with is another queue to be used as a signal. Each worker puts an item in the queue when its done. Then the printer keeps checking to see if it can pull enough signals from the done queue to equal the amount of workers launched. If so, it will exit.

This whole thing could still be rewritten better, but I am just applying bandaids to what you have. I sort of just threw this together with what you had.

Note, the way you start the whole process, checking for the directories under the starting path, will basically just exit out if there are only files.