Parallel processing in tidymodels w/ tune?

[josiah]

A blogpost from Nov. 2020 indicates that it is possible to do parallel processing of model training with the {tune} package.

The post says

"parallel_over . Setting this to a value of "resamples" will select this scheme to parallelize the computations."

However, when setting options(parallel_over = "resamples") or options("tune.parallel_over"="resamples") does not seem to enable parallel processing.

What is the best way to approach this?

Parallel processing with tune

[nirgrahamuk]

@josiah

does not seem to enable parallel processing

Perhaps you can say more about this?
Have you performed some form of benchmarking?

[hnagaty]

The post you refer to is only comparing different parallelization options, it doesn't explain how to enable it. You can refer to this post which explains how to enable parallel tuning. Optimizations and Parallel Processing • tune.

[josiah]

~That article tells the user how to register cores for parallel processing. It does not enable it, as far as I can tell. This is because you need to set the option parallel_over which is set to NULL by default. The question is where is this option set as it does not seem to be used from options(). I do not think this resolves the question, unfortunately.~

Edit: it comes from the control_grid()function and other control_*() in {tune}

[Max]

There is a lot more information in this section in tidy Models with R.

You need to load a parallel backend before calling the functions in tune. For example, you can use the doParallel package:

library(doParallel)
cl <- makePSOCKcluster(parallel::detectCores(logical = FALSE))
registerDoParallel(cl)

There are a number of do{technology} packages that can be used.