I just bought a new, machine to run python scripts for large scale modeling. It has two CPUs with 4 cores each (Xeon, 2.8GhZ). Each core has hyper-threading enabled for 4 logical cpu cores.
Now for the problem: When I run identical python processes in 8 separate terminals, top command shows that each process is taking 100% of the cpu. However, the process in terminal 1 is running about 4 times slower than the process in terminal 8. This seems odd to me…
I wonder if it has something to do with how the processes are scheduled on the various (logical?) cores? Does anyone have an idea of how I could all of the to run in about the same speed?
EDIT (in response to larsmans): Good point. The script is a ginat loop that runs about 10,000 times. Each loop reads in a text file (500 lines) and runs some basic calculations on the quantities read in. While the loop runs, it uses about 0.2% of Memory. There is no writing to disk during the loop. I could understand that the read access could be a limiting factor but I am perplexed about the fact that it would be the first process that would be the slowest if that was the case. I would have expected that it would get slower as I start more processes…
I timed the processes a couple of times using the time command in the terminal.
EDIT2: I just found out that sometimes a single core is designated to handle all reading and writing – so multiple processes (even if they run on separate cores) will use one single core for all the I/O… This would however only affect one of the cores, not cause all to have various processing speeds…
One reason for this might be the use of hyper-threading. HT logical CPUs appear to the operating system as separate CPUs, but really are not. So if two threads run on the same core in separate logical (HT) CPUs, performance would be smaller than if they ran on separate cores.
The easiest solution might be to disable hyper-threading. If that is not an option, use processor affinity to pin each Python process to its separate CPU.