Question

I asked a related question here and the response worked well: using parallel's parLapply: unable to access variables within parallel code

The problem is when I try to use the answer inside of the function it won’t work as I think it has to the default environment of clusterExport. I’ve read the vignette and looked at the help file but am approaching this with a very limited knowledge base. The way I used parLapply I expected it to behave similar to lapply but it doesn’t appear to.

Here is my attempt:

par.test <- function(text.var, gc.rate=10){ 
    ntv <- length(text.var)
    require(parallel)
    pos <-  function(i) {
        paste(sapply(strsplit(tolower(i), " "), nchar), collapse=" | ")
    }
    cl <- makeCluster(mc <- getOption("cl.cores", 4))
    clusterExport(cl=cl, varlist=c("text.var", "ntv", "gc.rate", "pos"))
    parLapply(cl, seq_len(ntv), function(i) {
            x <- pos(text.var[i])
            if (i%%gc.rate==0) gc()
            return(x)
        }
    )
}

par.test(rep("I like cake and ice cream so much!", 20))

#gives this error message
> par.test(rep("I like cake and ice cream so much!", 20))
Error in get(name, envir = envir) : object 'text.var' not found

Answer 1

By default clusterExport looks in the .GlobalEnv for objects to export that are named in varlist. If your objects are not in the .GlobalEnv, you must tell clusterExport in which environment it can find those objects.

You can change your clusterExport to the following (which I didn’t test, but you said works in the comments)

clusterExport(cl=cl, varlist=c("text.var", "ntv", "gc.rate", "pos"), envir=environment())

This way, it will look in the function’s environment for the objects to export.