Question

Background

I’m a bit new to developing and had a general python/programming question. If you have a method that is a recursion, what is involved to enabling multiple threads or multiprocessing? I’ve done some light reading and a few examples but they seem to be applying the syntax for new code(and not very cpu intensive tasks), I’m more wondering how do I re-design existing code to do this?

Say I have something thats cpu intensive(basically keeps adding to itself until limit is hit):

def adderExample(sum, number):
    if sum > 1000:
        print 'sum is larger than 10. Stoping'
    else:
        sum = sum + number
        print sum
        number = number + 1
        adderExample(sum, number)


adderExample(0,0)

Question(s)/Though process

How would I approach this to make it run faster assuming I have multiple cores available(I want it to eventually want it span machines but I think thats a sperate issue with hadoop so I’ll keep this example to only one system with multiple cpu’s)? It seems threading it isn’t the best choice(because of the time it takes to spawn new threads), if thats true should I only focus on multiprocessing? If so, can recursions be split to different cpu’s(vai queues I assume and then rejoin after its done)? Can I create multiple threads for each process than split those processes over multiple cpu’s? Lastly, is recursion depth limits an overall limit or is it based on threads/proceses, if so does multiprocessing/threading get around it?

Another question(related) how do those guys trying to codes(rsa, wireless keys,etc) via brute force overcome this problem? I assume they are scaling their mathematical processes over multiple cpu somehow. This or any example to build my understanding would be great.

Any tips/suggestions would be great

Thanks!

Answer 1

Such a loop wouldn’t benefit much at all from threading. Consider that you’re doing a series of additions, whose intermediate values depend on the previous iterations. This can’t be parallelized, because the threads would be stomping on each other’s values and overwriting things. You can lock the data so only one thread works on it at a time, but then you lose any benefit of having multiple threads working on that data.

Threads work best when they have independent data sets. e.g. a graphics renderer is a perfect example. Each thread renders a subset of the larger image – they may share common data sources for texture/vertex/color/etc… data, but each thread has its own little section of the total image to work one, and doesn’t touch other areas of the image. Whatever thread #1 does on its little section of pixels won’t affect what thread #2 is doing elsewhere in the image.

For your related question, password cracking is another example where threading/multiprocessing makes sense. Each thread goes off on its own testing multiple possible passwords against one common “to be cracked” list. What one thread is doing doesn’t affect any of the other cracker threads, unless you get a match, which may mean all threads abort since the job is “done”.

Once threads become interdependent on each other, you lose a lot of the benefits of having multiple threads. They’ll spend more time waiting for the other to finish than they’ll spend on doing actual work. Of course, this doesn’t say you should never use threads. Sometimes it does makes sense to have multiple threads, even if they are interdependent. E.g. a graphics thread + sound effects thread + action processor thread + A.I. calculations thread, etc… in a game. each one is nominally dependent on each other, but while the sound thread is busy generating the bang+ricochet audio for the gun the player just shot, the a.i. thread is off calculating what the game’s mobs are doing, the graphics thread is drawing some clouds in the background, etc…