Question

The snow package parXapply() functions distribute work very well when data is only contained in one list or matrix, but in this case I need to run a function on four different types of matrices.

For example, this is what I have now:

res.list = parLapply(cl, mynames, myfun, listA, listB, listC, listD)

myfun = function(name, listA, listB, listC, listD) {
  matrixA = listA[[name]]
  matrixB = listB[[name]]
  matrixC = listC[[name]]
  matrixD = listD[[name]]
}

The problem I am having is that the matrices are very large and I suspect calling parLapply() on the full lists involves transferring all the data to each cluster node. This can be very time-consuming and reduces the cluster performances.

How can I split the lists before calling myfun() and only send the relevant matrices to each node for processing?

Answer 1

clusterMap() does the job:

res.list = clusterMap(cl, myfun, mynames, listA, listB, listC, listD)

Somehow the parMapply() wrapper was left out of the package.