Generate timeout on caret::train()

Machine Learning and Modeling
1

Hi,

I tried to generate a timeout on caret::train() with R.utils::withTimeout on Windows without success. Indeed the computation runs to its term (~ 12 s) and then return a timeout (fixed at 5 s) error :

# Caret timeout ---------------------------------------------------------------
# Script to test timeout when training a caret model 
# -----------------------------------------------------------------------------
# Clear workspace -------------------------------------------------------------
rm(list = ls())
# Load library ----------------------------------------------------------------
library(dplyr)
# Load data -------------------------------------------------------------------
data("AirPassengers")
# Delete obs with NA ----------------------------------------------------------
data <- airquality %>% na.omit()
# Define trainControl ---------------------------------------------------------
trControl <- caret::trainControl(method = "LOOCV", allowParallel = FALSE)
# Train model -----------------------------------------------------------------
system.time(
  model <- R.utils::withTimeout({
    caret::train(Ozone ~ ., data = data, method = "nnet", trControl = trControl)
    # Sys.sleep(10)
  }, timeout = 5)
)

Error in { : task 2 failed - "reached CPU time limit"
Timing stopped at: 11.86 0.05 11.87

Thanks for any help not necessarily involving R.utils::withTimeout.

2

I can't exactly reproduce what you got since this should generate a lot more output.

My example is below via reprex. I added an option to shut off nnet's logging and upped the computations to last longer.

Also, train tracks execution time so the "Timing stopped" message is related to that.

library(dplyr)
#> 
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#> 
#>     filter, lag
#> The following objects are masked from 'package:base':
#> 
#>     intersect, setdiff, setequal, union
library(caret)
#> Loading required package: lattice
#> Loading required package: ggplot2
#> Registered S3 methods overwritten by 'ggplot2':
#>   method         from 
#>   [.quosures     rlang
#>   c.quosures     rlang
#>   print.quosures rlang
library(R.utils)
#> Loading required package: R.oo
#> Loading required package: R.methodsS3
#> R.methodsS3 v1.7.1 (2016-02-15) successfully loaded. See ?R.methodsS3 for help.
#> Registered S3 method overwritten by 'R.oo':
#>   method        from       
#>   throw.default R.methodsS3
#> R.oo v1.22.0 (2018-04-21) successfully loaded. See ?R.oo for help.
#> 
#> Attaching package: 'R.oo'
#> The following objects are masked from 'package:methods':
#> 
#>     getClasses, getMethods
#> The following objects are masked from 'package:base':
#> 
#>     attach, detach, gc, load, save
#> R.utils v2.8.0 successfully loaded. See ?R.utils for help.
#> 
#> Attaching package: 'R.utils'
#> The following object is masked from 'package:utils':
#> 
#>     timestamp
#> The following objects are masked from 'package:base':
#> 
#>     cat, commandArgs, getOption, inherits, isOpen, nullfile,
#>     parse, warnings

data("AirPassengers")

data <- airquality %>% na.omit()

trControl <- caret::trainControl(method = "LOOCV", allowParallel = FALSE)

system.time(
  model <- R.utils::withTimeout({
    caret::train(Ozone ~ ., data = data, method = "nnet", trControl = trControl,
                 # I added these options
                 trace = FALSE, tuneLength = 5)
  }, timeout = 5)
)
#> Warning: predictions failed for Fold065: size=7, decay=1e-04 Error in as.data.frame.matrix(newdata) : reached elapsed time limit
#> Error in pred - obs: non-numeric argument to binary operator
#> Timing stopped at: 8.104 0.118 8.223

sessionInfo()
#> R version 3.6.0 (2019-04-26)
#> Platform: x86_64-apple-darwin15.6.0 (64-bit)
#> Running under: macOS High Sierra 10.13.6
#> 
#> Matrix products: default
#> BLAS:   /Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libRblas.0.dylib
#> LAPACK: /Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libRlapack.dylib
#> 
#> locale:
#> [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
#> 
#> attached base packages:
#> [1] stats     graphics  grDevices utils     datasets  methods   base     
#> 
#> other attached packages:
#> [1] R.utils_2.8.0     R.oo_1.22.0       R.methodsS3_1.7.1 caret_6.0-82     
#> [5] ggplot2_3.1.1     lattice_0.20-38   dplyr_0.8.0.1    
#> 
#> loaded via a namespace (and not attached):
#>  [1] tidyselect_0.2.5   xfun_0.6           purrr_0.3.2       
#>  [4] reshape2_1.4.3     splines_3.6.0      colorspace_1.4-1  
#>  [7] generics_0.0.2     htmltools_0.3.6    stats4_3.6.0      
#> [10] yaml_2.2.0         survival_2.44-1.1  prodlim_2018.04.18
#> [13] rlang_0.3.4        ModelMetrics_1.2.2 pillar_1.3.1      
#> [16] glue_1.3.1         withr_2.1.2        foreach_1.4.4     
#> [19] plyr_1.8.4         lava_1.6.5         stringr_1.4.0     
#> [22] timeDate_3043.102  munsell_0.5.0      gtable_0.3.0      
#> [25] recipes_0.1.5      codetools_0.2-16   evaluate_0.13     
#> [28] knitr_1.22         class_7.3-15       highr_0.8         
#> [31] Rcpp_1.0.1         scales_1.0.0       ipred_0.9-8       
#> [34] digest_0.6.18      stringi_1.4.3      grid_3.6.0        
#> [37] tools_3.6.0        magrittr_1.5       lazyeval_0.2.2    
#> [40] tibble_2.1.1       crayon_1.3.4       pkgconfig_2.0.2   
#> [43] MASS_7.3-51.4      Matrix_1.2-17      data.table_1.12.0 
#> [46] lubridate_1.7.4    gower_0.2.0        assertthat_0.2.1  
#> [49] rmarkdown_1.12     iterators_1.0.10   R6_2.4.0          
#> [52] rpart_4.1-15       nnet_7.3-12        nlme_3.1-139      
#> [55] compiler_3.6.0

Created on 2019-05-15 by the reprex package (v0.2.1)

3

Hi Max, Thanks for replying !

Here is my output with your code example:

library(dplyr)
# Attachement du package : ‘dplyr’
# 
# The following objects are masked from ‘package:stats’:
#   
#   filter, lag
# 
# The following objects are masked from ‘package:base’:
#   
#   intersect, setdiff, setequal, union
# 
# Warning message:
#   le package ‘dplyr’ a Ă©tĂ© compilĂ© avec la version R 3.5.2 
library(caret)
# Le chargement a nécessité le package : lattice
# Le chargement a nécessité le package : ggplot2
library(R.utils)
# Le chargement a nécessité le package : R.oo
# Le chargement a nécessité le package : R.methodsS3
# R.methodsS3 v1.7.1 (2016-02-15) successfully loaded. See ?R.methodsS3 for help.
# R.oo v1.22.0 (2018-04-21) successfully loaded. See ?R.oo for help.
# 
# Attachement du package : ‘R.oo’
# 
# The following objects are masked from ‘package:methods’:
#   
#   getClasses, getMethods
# 
# The following objects are masked from ‘package:base’:
#   
#   attach, detach, gc, load, save
# 
# R.utils v2.8.0 successfully loaded. See ?R.utils for help.
# 
# Attachement du package : ‘R.utils’
# 
# The following object is masked from ‘package:utils’:
#   
#   timestamp
# 
# The following objects are masked from ‘package:base’:
#   
#   cat, commandArgs, getOption, inherits, isOpen, parse,
# warnings
# 
# Warning messages:
#   1: le package ‘R.utils’ a Ă©tĂ© compilĂ© avec la version R 3.5.3 
# 2: le package ‘R.oo’ a Ă©tĂ© compilĂ© avec la version R 3.5.2 
# 3: le package ‘R.methodsS3’ a Ă©tĂ© compilĂ© avec la version R 3.5.2 

data("AirPassengers")

data <- airquality %>% na.omit()

trControl <- caret::trainControl(method = "LOOCV", allowParallel = FALSE)

system.time(
  model <- R.utils::withTimeout({
    caret::train(Ozone ~ ., data = data, method = "nnet", trControl = trControl,
                 # I added these options
                 trace = FALSE, tuneLength = 5)
  }, timeout = 5)
)
# Error in pred - obs : non-numeric argument to binary operator
# In addition: Warning message:
#   model fit failed for Fold021: size=3, decay=1e-01 Error in .getXlevels(Terms, m) : reached elapsed time limit
# 
# Timing stopped at: 23.62 0.06 23.79

sessionInfo()
# R version 3.5.1 (2018-07-02)
# Platform: x86_64-w64-mingw32/x64 (64-bit)
# Running under: Windows >= 8 x64 (build 9200)
# 
# Matrix products: default
# 
# locale:
#   [1] LC_COLLATE=French_France.1252  LC_CTYPE=French_France.1252   
# [3] LC_MONETARY=French_France.1252 LC_NUMERIC=C                  
# [5] LC_TIME=French_France.1252    
# 
# attached base packages:
#   [1] stats     graphics  grDevices utils     datasets  methods   base     
# 
# other attached packages:
#   [1] R.utils_2.8.0     R.oo_1.22.0       R.methodsS3_1.7.1
# [4] caret_6.0-81      ggplot2_3.1.0     lattice_0.20-35  
# [7] dplyr_0.8.0.1    
# 
# loaded via a namespace (and not attached):
#   [1] Rcpp_1.0.0         gower_0.1.2        pillar_1.3.1      
# [4] compiler_3.5.1     plyr_1.8.4         class_7.3-14      
# [7] iterators_1.0.10   tools_3.5.1        rpart_4.1-13      
# [10] ipred_0.9-8        lubridate_1.7.4    tibble_2.0.1      
# [13] gtable_0.2.0       nlme_3.1-137       pkgconfig_2.0.2   
# [16] rlang_0.3.1        Matrix_1.2-14      foreach_1.4.4     
# [19] rstudioapi_0.8     yaml_2.2.0         prodlim_2018.04.18
# [22] withr_2.1.2        stringr_1.3.1      generics_0.0.2    
# [25] recipes_0.1.4      stats4_3.5.1       nnet_7.3-12       
# [28] grid_3.5.1         tidyselect_0.2.5   glue_1.3.0        
# [31] data.table_1.11.8  R6_2.3.0           survival_2.42-3   
# [34] lava_1.6.4         purrr_0.2.5        reshape2_1.4.3    
# [37] magrittr_1.5       splines_3.5.1      MASS_7.3-50       
# [40] scales_1.0.0       codetools_0.2-15   ModelMetrics_1.2.2
# [43] assertthat_0.2.0   timeDate_3043.102  colorspace_1.3-2  
# [46] stringi_1.2.4      lazyeval_0.2.1     munsell_0.5.0     
# [49] crayon_1.3.4

I conclude that there are two main differences between you and me:

Your config My config
mac os windows
R 3.6.0 R 3.5.1

I'll update my R version and see.

4

Hi Max,

I've updated my R version to 3.6.0 and the corresponding packages :

library(dplyr)
# Attachement du package : ‘dplyr’
# 
# The following objects are masked from ‘package:stats’:
#   
#   filter, lag
# 
# The following objects are masked from ‘package:base’:
#   
#   intersect, setdiff, setequal, union
library(caret)
# Le chargement a nécessité le package : lattice
# Le chargement a nécessité le package : ggplot2
# Registered S3 methods overwritten by 'ggplot2':
#   method         from 
# [.quosures     rlang
#   c.quosures     rlang
#   print.quosures rlang
#   Need help getting started? Try the cookbook for R:
#     http://www.cookbook-r.com/Graphs/
library(R.utils)
# Le chargement a nécessité le package : R.oo
# Le chargement a nécessité le package : R.methodsS3
# R.methodsS3 v1.7.1 (2016-02-15) successfully loaded. See ?R.methodsS3 for help.
# Registered S3 method overwritten by 'R.oo':
#   method        from       
# throw.default R.methodsS3
# R.oo v1.22.0 (2018-04-21) successfully loaded. See ?R.oo for help.
# 
# Attachement du package : ‘R.oo’
# 
# The following objects are masked from ‘package:methods’:
#   
#   getClasses, getMethods
# 
# The following objects are masked from ‘package:base’:
#   
#   attach, detach, gc, load, save
# 
# R.utils v2.8.0 successfully loaded. See ?R.utils for help.
# 
# Attachement du package : ‘R.utils’
# 
# The following object is masked from ‘package:utils’:
#   
#   timestamp
# 
# The following objects are masked from ‘package:base’:
#   
#   cat, commandArgs, getOption, inherits, isOpen, nullfile, parse,
# warnings

data("AirPassengers")
data <- airquality %>% na.omit()
trControl <- caret::trainControl(method = "LOOCV", allowParallel = FALSE)
system.time(
  model <- R.utils::withTimeout({
    caret::train(Ozone ~ ., data = data, method = "nnet", trControl = trControl,
                 # I added these options
                 trace = FALSE, tuneLength = 5)
  }, timeout = 5)
)
# Error in pred - obs : non-numeric argument to binary operator
# In addition: Warning message:
#   model fit failed for Fold021: size=5, decay=1e-01 Error in .deparseOpts(control) : reached elapsed time limit
# 
# Timing stopped at: 23.03 0.22 23.42
sessionInfo()
# R version 3.6.0 (2019-04-26)
# Platform: x86_64-w64-mingw32/x64 (64-bit)
# Running under: Windows >= 8 x64 (build 9200)
# 
# Matrix products: default
# 
# locale:
#   [1] LC_COLLATE=French_France.1252  LC_CTYPE=French_France.1252    LC_MONETARY=French_France.1252
# [4] LC_NUMERIC=C                   LC_TIME=French_France.1252    
# 
# attached base packages:
#   [1] stats     graphics  grDevices utils     datasets  methods   base     
# 
# other attached packages:
#   [1] R.utils_2.8.0     R.oo_1.22.0       R.methodsS3_1.7.1 caret_6.0-84      ggplot2_3.1.1    
# [6] lattice_0.20-38   dplyr_0.8.1      
# 
# loaded via a namespace (and not attached):
#   [1] Rcpp_1.0.1         pillar_1.4.0       compiler_3.6.0     gower_0.2.1       
# [5] plyr_1.8.4         iterators_1.0.10   class_7.3-15       tools_3.6.0       
# [9] rpart_4.1-15       ipred_0.9-9        lubridate_1.7.4    tibble_2.1.1      
# [13] nlme_3.1-139       gtable_0.3.0       pkgconfig_2.0.2    rlang_0.3.4       
# [17] Matrix_1.2-17      foreach_1.4.4      prodlim_2018.04.18 stringr_1.4.0     
# [21] withr_2.1.2        generics_0.0.2     recipes_0.1.5      stats4_3.6.0      
# [25] grid_3.6.0         nnet_7.3-12        tidyselect_0.2.5   data.table_1.12.2 
# [29] glue_1.3.1         R6_2.4.0           survival_2.44-1.1  lava_1.6.5        
# [33] reshape2_1.4.3     purrr_0.3.2        magrittr_1.5       ModelMetrics_1.2.2
# [37] scales_1.0.0       codetools_0.2-16   MASS_7.3-51.4      splines_3.6.0     
# [41] assertthat_0.2.1   timeDate_3043.102  colorspace_1.4-1   stringi_1.4.3     
# [45] lazyeval_0.2.2     munsell_0.5.0      crayon_1.3.4

Still not working. :disappointed_relieved:

I'll try on a linux machine instead of my windows computer to see.

5

Still not working :sweat: on a google virtual machine instance with debian 9 and R-3.6.0:


R version 3.6.0 (2019-04-26) -- "Planting of a Tree"
Copyright (C) 2019 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> library(dplyr)

Attaching package: ‘dplyr’

The following objects are masked from ‘package:stats’:

    filter, lag

The following objects are masked from ‘package:base’:

    intersect, setdiff, setequal, union

> library(caret)
Loading required package: lattice
Loading required package: ggplot2
Registered S3 methods overwritten by 'ggplot2':
  method         from 
  [.quosures     rlang
  c.quosures     rlang
  print.quosures rlang
> library(R.utils)
Loading required package: R.oo
Loading required package: R.methodsS3
R.methodsS3 v1.7.1 (2016-02-15) successfully loaded. See ?R.methodsS3 for help.
Registered S3 method overwritten by 'R.oo':
  method        from       
  throw.default R.methodsS3
R.oo v1.22.0 (2018-04-21) successfully loaded. See ?R.oo for help.

Attaching package: ‘R.oo’

The following objects are masked from ‘package:methods’:

    getClasses, getMethods

The following objects are masked from ‘package:base’:

    attach, detach, gc, load, save

R.utils v2.8.0 successfully loaded. See ?R.utils for help.

Attaching package: ‘R.utils’

The following object is masked from ‘package:utils’:

    timestamp

The following objects are masked from ‘package:base’:

    cat, commandArgs, getOption, inherits, isOpen, nullfile, parse,
    warnings

> data("AirPassengers")
> data <- airquality %>% na.omit()
> trControl <- caret::trainControl(method = "LOOCV", allowParallel = FALSE)
> system.time(
+ 	      model <- R.utils::withTimeout({
+ 		          caret::train(Ozone ~ .,
+ 				       data = data,
+ 				       method = "nnet",
+ 				       trControl = trControl,
+ 				       trace = FALSE, tuneLength = 5)
+ 			    }, timeout = 5)
+ 	      )
Error in pred - obs : non-numeric argument to binary operator
Calls: system.time ... tryCatchOne -> <Anonymous> -> throw -> throw.error -> <Anonymous>
In addition: Warning message:
model fit failed for Fold032: size=9, decay=1e-01 Error in lapply(m[xvars], function(x) if (is.factor(x)) levels(x) else if (is.character(x)) levels(as.factor(x))) : 
  reached elapsed time limit
 
Timing stopped at: 16.02 0.108 16.14
Execution halted
1 Like
6

EDIT - I hadn't refreshed the page so I missed your last post.

Can you run it without the interrupt and make sure that is wouldn't fail otherwise on this system? I don't know why it would but...

caret uses foreach to run these computations and, even though you don't appear to be running in parallel, it might be related to how foreach works on this system.

7

Hi Max, back on my computer with windows I've decided to make 4 tests:

  • Test 1 : no parallel backend | no timeout
  • Test 2 : no parallel backend | timeout
  • Test 3 : parallel backend | no timeout
  • Test 4 : parallel backend | timeout
# Clear workspace -------------------------------------------------------------
rm(list = ls())

# Load library ----------------------------------------------------------------
library(dplyr)
#> 
#> Attachement du package : 'dplyr'
#> The following objects are masked from 'package:stats':
#> 
#>     filter, lag
#> The following objects are masked from 'package:base':
#> 
#>     intersect, setdiff, setequal, union
library(caret)
#> Le chargement a nécessité le package : lattice
#> Le chargement a nécessité le package : ggplot2
#> Registered S3 methods overwritten by 'ggplot2':
#>   method         from 
#>   [.quosures     rlang
#>   c.quosures     rlang
#>   print.quosures rlang
library(R.utils)
#> Le chargement a nécessité le package : R.oo
#> Le chargement a nécessité le package : R.methodsS3
#> R.methodsS3 v1.7.1 (2016-02-15) successfully loaded. See ?R.methodsS3 for help.
#> Registered S3 method overwritten by 'R.oo':
#>   method        from       
#>   throw.default R.methodsS3
#> R.oo v1.22.0 (2018-04-21) successfully loaded. See ?R.oo for help.
#> 
#> Attachement du package : 'R.oo'
#> The following objects are masked from 'package:methods':
#> 
#>     getClasses, getMethods
#> The following objects are masked from 'package:base':
#> 
#>     attach, detach, gc, load, save
#> R.utils v2.8.0 successfully loaded. See ?R.utils for help.
#> 
#> Attachement du package : 'R.utils'
#> The following object is masked from 'package:utils':
#> 
#>     timestamp
#> The following objects are masked from 'package:base':
#> 
#>     cat, commandArgs, getOption, inherits, isOpen, nullfile,
#>     parse, warnings

# Load data -------------------------------------------------------------------
data("AirPassengers")
data <- airquality %>% na.omit()

# Parameters ------------------------------------------------------------------
nbCore <- parallel::detectCores() - 1

# Test 1 : no parallel | no timeout
trControl <- caret::trainControl(method = "LOOCV", allowParallel = FALSE)
system.time(
  model <- caret::train(Ozone ~ ., 
                        data = data,
                        method = "nnet",
                        trControl = trControl,
                        trace = FALSE, tuneLength = 10)
)
#> utilisateur     systÚme      écoulé 
#>       66.50        0.03       66.54

# Test 2 : no parallel | timeout
trControl <- caret::trainControl(method = "LOOCV", allowParallel = FALSE)
system.time(
  model <- R.utils::withTimeout({
    caret::train(Ozone ~ ., 
                 data = data,
                 method = "nnet",
                 trControl = trControl,
                 trace = FALSE, tuneLength = 10)
  }, timeout = 5)
)
#> Warning: predictions failed for Fold009: size= 1, decay=0.1000000000 Error in `[.data.frame`(object, !omit, , drop = FALSE) : 
#>   la limite de temps est atteinte
#> Error in pred - obs: argument non numérique pour un opérateur binaire
#> Timing stopped at: 67.64 0.03 67.74

# Test 3 : parallel    | no timeout
doParallel::registerDoParallel(nbCore)
showConnections()
#>    description               class            mode  text     isopen  
#> 4  "output"                  "textConnection" "wr"  "text"   "opened"
#> 5  "<-DESKTOP-2BU44RT:11895" "sockconn"       "a+b" "binary" "opened"
#> 6  "<-DESKTOP-2BU44RT:11895" "sockconn"       "a+b" "binary" "opened"
#> 7  "<-DESKTOP-2BU44RT:11895" "sockconn"       "a+b" "binary" "opened"
#> 8  "<-DESKTOP-2BU44RT:11895" "sockconn"       "a+b" "binary" "opened"
#> 9  "<-DESKTOP-2BU44RT:11895" "sockconn"       "a+b" "binary" "opened"
#> 10 "<-DESKTOP-2BU44RT:11895" "sockconn"       "a+b" "binary" "opened"
#> 11 "<-DESKTOP-2BU44RT:11895" "sockconn"       "a+b" "binary" "opened"
#>    can read can write
#> 4  "no"     "yes"    
#> 5  "yes"    "yes"    
#> 6  "yes"    "yes"    
#> 7  "yes"    "yes"    
#> 8  "yes"    "yes"    
#> 9  "yes"    "yes"    
#> 10 "yes"    "yes"    
#> 11 "yes"    "yes"
trControl <- caret::trainControl(method = "LOOCV", allowParallel = TRUE)
system.time(
  model <- caret::train(Ozone ~ ., 
                        data = data,
                        method = "nnet",
                        trControl = trControl,
                        trace = FALSE, tuneLength = 10)
)
#> utilisateur     systÚme      écoulé 
#>        6.64        0.62       21.25
doParallel::stopImplicitCluster()
showConnections()
#>   description class            mode text   isopen   can read can write
#> 4 "output"    "textConnection" "wr" "text" "opened" "no"     "yes"

# Test 4 : parallel    | timeout
doParallel::registerDoParallel(nbCore)
showConnections()
#>    description               class            mode  text     isopen  
#> 4  "output"                  "textConnection" "wr"  "text"   "opened"
#> 5  "<-DESKTOP-2BU44RT:11895" "sockconn"       "a+b" "binary" "opened"
#> 6  "<-DESKTOP-2BU44RT:11895" "sockconn"       "a+b" "binary" "opened"
#> 7  "<-DESKTOP-2BU44RT:11895" "sockconn"       "a+b" "binary" "opened"
#> 8  "<-DESKTOP-2BU44RT:11895" "sockconn"       "a+b" "binary" "opened"
#> 9  "<-DESKTOP-2BU44RT:11895" "sockconn"       "a+b" "binary" "opened"
#> 10 "<-DESKTOP-2BU44RT:11895" "sockconn"       "a+b" "binary" "opened"
#> 11 "<-DESKTOP-2BU44RT:11895" "sockconn"       "a+b" "binary" "opened"
#>    can read can write
#> 4  "no"     "yes"    
#> 5  "yes"    "yes"    
#> 6  "yes"    "yes"    
#> 7  "yes"    "yes"    
#> 8  "yes"    "yes"    
#> 9  "yes"    "yes"    
#> 10 "yes"    "yes"    
#> 11 "yes"    "yes"
trControl <- caret::trainControl(method = "LOOCV", allowParallel = TRUE)
system.time(
  model <- R.utils::withTimeout({
    caret::train(Ozone ~ ., 
                 data = data,
                 method = "nnet",
                 trControl = trControl,
                 trace = FALSE, tuneLength = 10)
  }, timeout = 5)
)
#> [2019-05-17 14:42:22] TimeoutException: la limite de temps est atteinte
#> [cpu=5s, elapsed=5s]
#> Timing stopped at: 1.5 0.04 5.04
doParallel::stopImplicitCluster()
showConnections()
#>   description class            mode text   isopen   can read can write
#> 4 "output"    "textConnection" "wr" "text" "opened" "no"     "yes"

# Session info ----------------------------------------------------------------
sessionInfo()
#> R version 3.6.0 (2019-04-26)
#> Platform: x86_64-w64-mingw32/x64 (64-bit)
#> Running under: Windows 10 x64 (build 17763)
#> 
#> Matrix products: default
#> 
#> locale:
#> [1] LC_COLLATE=French_France.1252  LC_CTYPE=French_France.1252   
#> [3] LC_MONETARY=French_France.1252 LC_NUMERIC=C                  
#> [5] LC_TIME=French_France.1252    
#> 
#> attached base packages:
#> [1] stats     graphics  grDevices utils     datasets  methods   base     
#> 
#> other attached packages:
#> [1] R.utils_2.8.0     R.oo_1.22.0       R.methodsS3_1.7.1 caret_6.0-84     
#> [5] ggplot2_3.1.1     lattice_0.20-38   dplyr_0.8.1      
#> 
#> loaded via a namespace (and not attached):
#>  [1] tidyselect_0.2.5   xfun_0.7           purrr_0.3.2       
#>  [4] reshape2_1.4.3     splines_3.6.0      colorspace_1.4-1  
#>  [7] generics_0.0.2     htmltools_0.3.6    stats4_3.6.0      
#> [10] yaml_2.2.0         survival_2.44-1.1  prodlim_2018.04.18
#> [13] rlang_0.3.4        ModelMetrics_1.2.2 pillar_1.4.0      
#> [16] glue_1.3.1         withr_2.1.2        foreach_1.4.4     
#> [19] plyr_1.8.4         lava_1.6.5         stringr_1.4.0     
#> [22] timeDate_3043.102  munsell_0.5.0      gtable_0.3.0      
#> [25] recipes_0.1.5      codetools_0.2-16   evaluate_0.13     
#> [28] knitr_1.22         doParallel_1.0.14  parallel_3.6.0    
#> [31] class_7.3-15       highr_0.8          Rcpp_1.0.1        
#> [34] scales_1.0.0       ipred_0.9-9        digest_0.6.18     
#> [37] stringi_1.4.3      grid_3.6.0         tools_3.6.0       
#> [40] magrittr_1.5       lazyeval_0.2.2     tibble_2.1.1      
#> [43] crayon_1.3.4       pkgconfig_2.0.2    MASS_7.3-51.4     
#> [46] Matrix_1.2-17      data.table_1.12.2  lubridate_1.7.4   
#> [49] gower_0.2.1        assertthat_0.2.1   rmarkdown_1.12    
#> [52] iterators_1.0.10   R6_2.4.0           rpart_4.1-15      
#> [55] nnet_7.3-12        nlme_3.1-139       compiler_3.6.0

Results:

parallel backend timeout (5s) time connection
no no 66s none
no yes 67s none
yes no 21s 7
yes yes 5s 7

So only when a parallel backend has been created the timeout works.

Nota:

  • I use the function showConnections() in order to see if the workers are correctly closed.
  • Next step for me is to make caret::train() works properly with a timeout inside a tryCatch()
2 Likes
8

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.