Shouldn't batch size in grid_latin_hypercube() be a power of 2 (2^x)

[vidaringa]

I read over and over again that when tuning a Neural Network the batch size should, preferably, be power of 2 due to memory allocations. I.e. 2^10, 2^11 etc.

Is there a reason why this is not done by default when creating values for batch_size using a function like grid_latin_hypercube() ?

[Max]

The log-2 thing is just a convenience for certain types of worker processes that are available to do the computations (see this SO post).

It can be anything so we leave it as-is. You can certainly transform it though

library(tidymodels)
tidymodels_prefer()

set.seed(1)
parameters(batch_size(c(1L, 15L)), penalty()) %>% 
  grid_latin_hypercube(size = 10) %>% 
  mutate(
    rounded = 2^round(log2(batch_size), 0),
    log2_value = log2(rounded)
  )
#> # A tibble: 10 × 4
#>    batch_size  penalty rounded log2_value
#>         <int>    <dbl>   <dbl>      <dbl>
#>  1        599 8.95e- 2     512          9
#>  2         55 4.45e- 3      64          6
#>  3         30 4.34e- 8      32          5
#>  4       3769 2.37e- 4    4096         12
#>  5        130 8.84e-10     128          7
#>  6      10565 5.02e- 7    8192         13
#>  7       1426 3.13e- 5    1024         10
#>  8          2 4.50e- 6       2          1
#>  9          6 5.12e- 9       8          3
#> 10      32482 2.43e- 1   32768         15

^{Created on 2022-01-30 by the reprex package (v2.0.1)}

[vidaringa]

Hi Max,

Aweseom, thank you for clarifying