parLapply in nested function

parallel

#1

I have a script like the one below, with a function called within its own definition - using a cluster object cl. The problem is that each iteration of the first for loop will run in parallel, whereas the same computation within the if statement will use only 1 core. There's no error or warning, it just uses one core.

Unfortunately this MWE doesn't reproduce the problem, its too tricky for me to reproduce at this stage. I'll try to fix it myself, I was just wondering whether anyone had encountered this before or knows whether there's a rule against parallelism in nested functions?

cl <- parallel::makeCluster(detectCores())

prod_exp <- function(x){prod(log((exp(x)))}

MWE <- function(cl, y, threshold){
  
  lfunc <- function(...){parallel::parLapply(cl, ...)}
  
  for(i in 1:10){
    y <- y * as.numeric(lfunc(X = rnorm(1e6), fun = prod_exp))
  }
  if(y < threshold){
    print("resampling")
    y <- MWE(cl, y, -Inf)
  }
  return(y)
}

MWE(cl, 3, 5)

Thanks for your help.


#2

Nevermind, I sorted it out. The input differed slightly, slowing down the nested function.


#3

If your question's been answered (even if by you), would you mind choosing a solution? (See FAQ below for how).

Having questions checked as resolved makes it a bit easier to navigate the site visually and see which threads still need help.

Thanks


#4

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.