Question

I am writing a simple python code for a very task which some of the best minds i guess are working on? Anyways.
I have a really powerful desktop 8cores (16 virtual cores).
I want to write a program whcih can find the distinct words in the whole corpus of words.
Or think of other task like things like word frequency counts.
Though map-reduce stuff works great for distributed framework.
Is there a way to make use of all the cores of your processor? that is multicore execution of code.

Or maybe this.
if i have to do the following:

def hello_word():
   print "hello world!"

and instead of python hello_world.py
i want to run this hello_world.py using all the cores of my processor.
what changes will i make.?
Thanks

Answer 1

You first need to identify how your process can be broken down into parallel pieces. Running your example in your question on multiple cores makes absolutely no sense because there is just a single task to accomplish and no way it can be broken down into simpler, parallel steps.

After you have figured out how to break your task into parallel pieces, take a look at the multiprocessing module as Michael mentioned in comments. Working through some of the examples on that page is a good way to start.