Possible Duplicate:
Parallel processing in R limited
I’ve written some code in R multicore, and I’m running it on a 24-core machine. In fact there are only 12 cores, but they are hyperthreaded, so it looks like there are 24.
Here’s what’s strange: all the threads run on the same single core! So they each only use a tiny amount of cpu, instead of each running on a single core, and chewing up all available cores.
For simplicity, I’m just running 4 threads:
mclapply( 1:30, function(size) {
# time consuming stuff that is cpu bound (think "forecast.ets" et al)
}, mc.cores = 4, mc.preschedule = F )
Prior to running this, there is already an R process running on one core, using 100% of that core’s capacity:
Next, I launch the “multicore process”, and 4 extra threads fight for the same core!:
… so, they each get 12% of one core, or about 1% of the available processing power, when they should each be able to get 100% of one core. Also, the other R process now only get 50% of the core.
OS is Ubuntu 12.04 64-bit. Hardware is Intel. R is version 2.15.2 “trick or treat”
Thoughts? (I know I could just use snowfall, but I have a ton of variables, and I really don’t want to have to sfExport all of them!)
Edit: oh, I guess there’s some global lock somewhere? But still, why would there be a conflict between two completely separate R processes? I can run two R processes in parallel just fine, with each taking 100% of a core’s CPU.
Edit2: Thanks to Dirk’s pointer, I rebuilt openblas, and it’s looking much healthier now!:
A possible issue is a possible side effect of the OpenBLAS package which sets CPU affinity such that processes stick to one core. See Parallel processing in R limited for a discussion and link to more discussion on the r-sig-hpc list which has a fix.