tfrun on Keras Hyperparameter Tuning question

[Tony_Duan]

Hi.
I am using tfrun on Keras Hyperparameter Tuning.

For example,following code is to test dense_units1=c(64,128,256) successfully

But how to test the model without dense_units1? dense_units1=c(0,64,128,256) will not work.

tfruns::tuning_run('number_recognition_002_model.R',
                     #sample=0.1, # use 10%
                     flags = list(
                       dense_units1=c(64,128,256),
                       dense_units2=c(64,128,256),
                       dropout1=c(0.1,0.2),
                       dropout2=c(0.1)
                       
                       #batch_size=c(32,64)
                       )
)

number_recognition_002_model.R:

FLAGS <- flags(
  flag_numeric("dense_units1", 64),
  flag_numeric("dense_units2", 64),
  flag_numeric("dropout1", 0.4),
  flag_numeric("dropout2", 0.3)
 
)

model <- keras_model_sequential() 
model %>% 
  layer_dense(units = FLAGS$dense_units1, activation = "relu", input_shape = c(784)) %>% 
  layer_dropout(rate = FLAGS$dropout1) %>% 
  layer_dense(units = FLAGS$dense_units2, activation = "relu") %>%
  layer_dropout(rate = FLAGS$dropout2) %>%
  layer_dense(units = 10, activation = "softmax")


summary(model)

Thank you

[dfalbel]

Hi Tony,

I would do something like this in the model code:

model <- keras_model_sequential() 

if (FLAGS$dense_units1 != 0) {
 model %>%
   layer_dense(units = FLAGS$dense_units1, activation = "relu", input_shape = c(784)) %>% 
   layer_dropout(rate = FLAGS$dropout1)
}
 
model %>%
  layer_dense(units = FLAGS$dense_units2, activation = "relu") %>%
  layer_dropout(rate = FLAGS$dropout2) %>%
  layer_dense(units = 10, activation = "softmax")

Ie. You can modify the model definition code based on values of the FLAGS.

[Andrea]

Shouldn't it be

model <- keras_model_sequential() 

if (FLAGS$dense_units1 != 0) {
 model %>%
   layer_dense(units = FLAGS$dense_units1, activation = "relu", input_shape = c(784)) %>% 
   layer_dropout(rate = FLAGS$dropout1)  
}

etc.? I.e., shouldn't you remove the last pipe from the code in the if clause?

Anyway, your suggestion is very good, but to help @Tony_Duan understand why his approach doesn't work, I just wanted to note that without dense_units1, his model would lack an input layer:

model %>% 
  layer_dropout(rate = FLAGS$dropout1) %>% 
  layer_dense(units = FLAGS$dense_units2, activation = "relu") %>%
  layer_dropout(rate = FLAGS$dropout2) %>%
  layer_dense(units = 10, activation = "softmax")

This model wouldn't make sense because a dropout layer cannot be the input layer. This is why, in your alternative model, you have to keep these instruction together in the if clause:

   layer_dense(units = FLAGS$dense_units1, activation = "relu", input_shape = c(784)) %>% 
   layer_dropout(rate = FLAGS$dropout1)  

[dfalbel]

You right. Just edited my answer.

Also, the problem is that you can't have a layer_dense with 0 units. So you need to create an auxiliary code to make sure the layer is not included in the model when you send a 0 as hyperparameter.

[Tony_Duan]

Thanks both. But if I just include 0 in dense_units2 .it will also fail.

So it seems that I need 2 if function for 2 dense_units

tfruns::tuning_run('number_recognition_002_model.R',
                     #sample=0.1, # use 10%
                     flags = list(
                       dense_units1=c(64,128,256),
                       dense_units2=c(0,64,128,256),
                       dropout1=c(0.1,0.2),
                       dropout2=c(0.1)
                       
                       #batch_size=c(32,64)
                       )
)

[Andrea]

Hi, Tony,

sure! As Daniel said,

@dfalbel

Also, the problem is that you can't have a layer_dense with 0 units.

so, this is valid for all dense layer, irrespective of whether they're 1 or 2.