I've searched the web extensively and have tried to troubleshoot this for a few months, but I'm at my wits end!
In a variety of R packages that have an option for parallelization, I am unable to disable parallel core usage (or set the number of cores to be used).
I first noticed this when running the DESeq2 package, which uses the BiocParallel package for parallel computing (post here). I would see a large number of my cores engaged despite registering my serial backend. Even when I explicitly registered a parallel backend and set the number of workers to 1 or 2, this is ignored and many cores get engaged.
Recently, I have been using the caret package for machine learning and ran into this problem again.
As a minimal example, I ran this in a fresh R session and it engages 65 cores.
library(caret) random_matrix <- matrix(rnorm(1e6), nrow = 1000, ncol = 1000) random_df <- as.data.frame(random_matrix) random_df$class <- as.factor(rbinom(1000, 1, 0.5)) training <- train(class ~ ., data = random_df, method = 'glm')
Following caret's guide for using parallel processing, doParallel, foreach, or doMC are all compatible. I follow the given example and use doParallel.
However, even when I run the following code to explicitly set 2 workers, it still engages 65 cores.
This happens even when I pass in the parameter allowParallel = FALSE.
library(caret) library(doParallel) cl <- makePSOCKcluster(2) # 2 workers registerDoParallel(cl) random_matrix <- matrix(rnorm(1e6), nrow = 1000, ncol = 1000) random_df <- as.data.frame(random_matrix) random_df$class <- as.factor(rbinom(1000, 1, 0.5)) training <- train(class ~ ., data = random_df, method = 'glm', trControl = trainControl(allowParallel = FALSE)) )
I don't think this issue relates to the caret package specifically, since I have run into this problem in a completely unrelated package (e.g. DESeq2).
I'm wondering if somewhere there has been a parallel backend "secretly" registered for 65 workers that I'm unaware of. I am at a complete loss for trying to find it and set it properly. I have tried explicitly setting the workers in BiocParallel, future, foreach, doParallel, and doMC but nothing seems to make a difference.
I also wonder if it may relate to this: https://github.com/HenrikBengtsson/Wishlist-for-R/issues/94
How would I go about tracking down this problem? Is there a way to completely reset my parallel parameters?
If it helps, I am running R 3.6.3 on an Ubuntu 18.04 server. Here is my sessionInfo()
R version 3.6.3 (2020-02-29) Platform: x86_64-pc-linux-gnu (64-bit) Running under: Ubuntu 18.04.4 LTS Matrix products: default BLAS: /usr/lib/x86_64-linux-gnu/openblas/libblas.so.3 LAPACK: /usr/lib/x86_64-linux-gnu/libopenblasp-r0.2.20.so locale:  LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 LC_PAPER=en_US.UTF-8  LC_NAME=C LC_ADDRESS=C LC_TELEPHONE=C LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C attached base packages:  parallel stats graphics grDevices utils datasets methods base other attached packages:  doParallel_1.0.15 iterators_1.0.12 foreach_1.5.0 caret_6.0-86 ggplot2_3.3.0 lattice_0.20-41 loaded via a namespace (and not attached):  rstan_2.19.3 tidyselect_1.0.0 xfun_0.13 purrr_0.3.4 reshape2_1.4.4 splines_3.6.3 colorspace_1.4-1 vctrs_0.2.4 generics_0.0.2  stats4_3.6.3 loo_2.2.0 survival_3.1-12 prodlim_2019.11.13 rlang_0.4.6 pkgbuild_1.0.7 ModelMetrics_126.96.36.199 pillar_1.4.3 glue_1.4.0  withr_2.2.0 matrixStats_0.56.0 lifecycle_0.2.0 plyr_1.8.6 lava_1.6.7 stringr_1.4.0 timeDate_3043.102 munsell_0.5.0 gtable_0.3.0  recipes_0.1.12 codetools_0.2-16 inline_0.3.15 knitr_1.28 callr_3.4.3 ps_1.3.2 class_7.3-17 fansi_0.4.1 Rcpp_188.8.131.52  scales_1.1.0 ipred_0.9-9 StanHeaders_2.19.2 gridExtra_2.3 stringi_1.4.6 processx_3.4.2 dplyr_0.8.5 grid_3.6.3 cli_2.0.2  tools_3.6.3 magrittr_1.5 tibble_3.0.1 crayon_1.3.4 pkgconfig_2.0.3 ellipsis_0.3.0 MASS_7.3-51.6 Matrix_1.2-18 prettyunits_1.1.1  data.table_1.12.8 pROC_1.16.2 lubridate_1.7.8 gower_0.2.1 assertthat_0.2.1 rstudioapi_0.11 R6_2.4.1 rpart_4.1-15 nnet_7.3-14  nlme_3.1-147 compiler_3.6.3