This a follow-up to Parallelize function taking external pointers (XPtr)
I won't reproduce the Cpp code here to make things shorter. The problem was to that once a parameter of a function is evaluated, it is defined in the function's environment and, in the case of an external pointer, is no longer available in a fork cluster.
So while this function worked:
require(parallel)
test1 <- function(a) {
cl <- makeForkCluster(nnodes=2)
r <- parLapply(cl, 1:5, function(i) g(a,i) )
stopCluster(cl)
unlist(r)
}
This function didn't:
test2 <- function(a) {
cl <- makeForkCluster(nnodes=2)
p <- g(a, 0)
r <- parLapply(cl, 1:5, function(i) g(a,i) )
stopCluster(cl)
unlist(r)
}
As pointed out by Ralf Stubner, this came from the fact that the call g(a, 0)
forced the evaluation of the promise a
. He suggested the following work around (here with two debug prints to understand how it works):
test3 <- function(a) {
cl <- makeForkCluster(nnodes = 2)
print(pryr::promise_info(a))
b <- eval(substitute(a))
p <- g(b, 0)
print(pryr::promise_info(a))
r <- parLapply(cl, 1:5, function(i) g(a,i) )
stopCluster(cl)
unlist(r)
}
This allowed to access to whatever was in a
, but a
was still an unevaluated promise. But this doesn't work when test3
is called from another function!
test4 <- function(b) test1(b)
test5 <- function(b) test3(b)
While test4
works well (the recursive promise evaluation seems to give a valid pointer), the workaround in test3
no longer works when called from test5
.
The debugging prints show that despite the eval(substitute(a))
trick, the promise is evaluated. My understanding is that this tricks forces the evaluation of the promise b
of test5
in its environment, thus a
becomes evaluated too.
Is there another workaround? (I tried to play with pryr::parent_promise
but even the code from the example in the man page gives strange results).
I have other complex problems of this type. A general way to get the content of a promise without evaluating it, or to pass external pointers to other functions, with a parLapply
call at the very end, without stumbling constantly on this problem, would be much welcome.