Question

I am currently working in Python and my program looks like this:

function(1)
function(2)
...
function(100)

Performing a function takes ~30 minutes at 100% CPU, so executing the program takes a lot of time. The functions access the same file for inputs, do a lot of math and print the results.

Would introducing multithreading decrease the time, which the program takes to complete (I am working on a multicore machine)? If so, how many threads should I use?

Thank you!

Answer 1

It depends.

If none of the functions depend on each other at all, you can of course run them on separate threads (or even processes using multiprocessing, to avoid the global interpreter lock). You can either run one process per core, or run 100 processes, or any number in between, depending on the resource constraints of your system. (If you don’t own the system, some admins don’t like users who spam the process table.)

If the functions must be run one after the other, then you can’t do that. You have to restructure the program to try and isolate independent tasks, or accept that you might have a P-complete (inherently hard to parallelize) problem and move on.