replace `mclapply` with `future_lapply`

Hi all,

I'm looking for general advice relating to modifying an existing function that uses mclapply for parallel computing across many cores and changing it to use a relevant futureverse function that can distribute the work across nodes in a cluster.

The function in question has a lot of moving parts (linked below in case anyone is interested) because the function being applied to list elements calls unexported functions defined within other function definitions. Just swapping the parallelization funs didn't work and I'm still trying to understand why.

I tried to replicate the errors I keep getting but can't seem to break future_lapply with nested fn definitions, packages, and unexported functions. Like so:

library(future.apply)
#> Loading required package: future
library(dplyr)
#> 
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#> 
#>     filter, lag
#> The following objects are masked from 'package:base':
#> 
#>     intersect, setdiff, setequal, union
library(usethis)

iris <- datasets::iris
chunks <- split(iris, seq(1, nrow(iris), length.out = 5L))


# define fns
mult_length <- function(vec){vec*3}
get_min1 <- function(z){min(z)+1}

# with nested definitions and unexported fns
sumlength <- function(x,constantval=3){
  drop_col3 <- function(dat){dplyr::select(dat,-1)} # nested fn
  newdat <- drop_col3(x)
  length_vec <- sum(newdat[,1])
  mult_length <- function(y){y*3} # another nested fn
  multiplied <- mult_length(length_vec)
  usethis:::base_and_recommended() # unexported fn
  get_min1(multiplied)+constantval
}

sumlength(iris)
#> [1] 1379.8
future_lapply(chunks, FUN = sumlength) # works OK
#> $`1`
#> [1] 287.2
#> 
#> $`38.25`
#> [1] 281.8
#> 
#> $`75.5`
#> [1] 276.4
#> 
#> $`112.75`
#> [1] 270.4
#> 
#> $`150`
#> [1] 280

Created on 2022-03-15 by the reprex package (v2.0.1)
I will try to break down the process into smaller bits, but in the meantime, if someone here has successfully replaced mclapply with anything from future.apply please let me know, I'd appreciate any feedback.
Thanks!

The fns in question:

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.

If you have a query related to it or one of the replies, start a new topic and refer back with a link.