Question

I have this code and have noticed when specifying more than 1 thread, performance suffers. The concept here is simple, setup a Queue in multiprocessing, access it from each thread, equally– note I am using an easily divisible queue size here.

import time
from  multiprocessing import Process, Queue

def worker(q,x):
    for i in range(x):
        message = q.get()


if __name__ == "__main__":

    workers=[]
    threads=1
    x = int(1000000/threads) #break up the work for each thread
    q = Queue()

    for i in range(threads):
        p=Process(target=worker, args=(q,x))
        workers.append(p)


    print('creating queue')
    for num in range(1000000):
        q.put("MESSAGE")

    start_time = time.time()

    print('starting jobs')
    for p in workers:
        p.start()

    print('waiting until jobs finish')
    for p in workers:
        p.join()

    end_time = time.time()

    duration = end_time - start_time
    msg_per_sec = 1000000 / duration

    print ("Duration: %s" % duration)
    print ("Messages Per Second: %s" % msg_per_sec)

I have made a different script with multiprocessing threads performing urllib requests and it shows that scaling processors helps speed things up, so what am I doing wrong here; is it that I am accessing queue from all of the processes and am essentially blocking?

Answer 1

The issue is that your workers are not doing any useful work. All the time is spent getting stuff off the queue. The queue is shared and all access to it is serialized, hence no speedup.

Once the workers start taking some CPU or I/O time to process each message, things should start improving.

See Amdahl’s Law.