Question

I am trying to speed up my code with a ‘foreach’ loop using the doSMP package.

Here is a simplified version of my issue: I am running a file called main.R

file: main.R:

require(doSMP)
dropbox_path = "/home/ruser/Dropbox"
workers <- startWorkers(4)
registerDoSMP(workers)
foreach(jj=1:4 ) %dopar% source("test.R")
stopWorkers(workers)

file: test.R:

message(dropbox_path)

This returns the following error:
“Error in source(“test.R”) :
task 1 failed – “object ‘dropbox_path’ not found”

If I modify main.R to be :

require(doSMP)
dropbox_path = "/home/ruser/Dropbox"
workers <- startWorkers(4)
registerDoSMP(workers)
foreach(jj=1:4 ) %dopar% message(dropbox_path)
stopWorkers(workers)

It then works very well.
It also used to work well with sequencial code (‘for’ instead of ‘foreach’).

So R child instances can access the dropbox_path variable, but not when it is parsed through the source function. I tried to play around with the source() function arguments ‘local’ and ‘chdir’ with no sucess.

Would you know a way for the code to work? I would like to keep using the source() function.

Answer 1

I apologize in advance that my little workaround doesn’t use your tools, but here’s how I would do it. I use package snowfall because I can easily extend my apply functions to work on multiple cores. The code is not tested because all of my cores are currently occupied. Should work, though.

tiny_script.R contents:

date()

R code to fire up multiple cores:

library(snowfall)
sfInit(parallel = TRUE, cpus = 4, type = "SOCK") #power up
my.list <- vector("list", 10)

sfLapply(x = my.list, fun = function(x) source("./Odpad/tiny_script.R")) 

Running on a single core using lapply only:

> lapply(X = my.list, FUN = function(x) source("./Odpad/tiny_script.R")) #notice the difference in argument names between `lapply` and `sfLapply`.
[[1]]
[[1]]$value
[1] "Wed Jun 22 13:02:11 2011"

[[1]]$visible
[1] TRUE


[[2]]
[[2]]$value
[1] "Wed Jun 22 13:02:11 2011"