use AUC as metric in keras for R

[Kevin]

Hi,

I was wondering whether there is a convenient way of using ROC AUC instead of accuracy when compile-ing a keras model in R. AUC does not seem to be in the set of standard measures...

Thanks,

Kevin

[mattwarkentin]

Hi Kevin,

You basically have two options for using AUC with keras:

1. Build a custom metric - this can be done using the keras::custom_metric() function (and there are a few other helper functions). The function can accept y_true and y_pred as arguments, but these two arguments will be tensors so you'll have to use back-end tensor functions to perform any calculations. This is kind of annoying because you can't use existing R functions for computing ROC/AUC like yardstick::roc_auc() or pROC::roc().

2. Build a custom callback function - this is a simpler way of getting AUC during each batch/epoch, but this is really just for reporting/saving AUCs. The AUC isn't used for making training decision such as early stopping. You can use callback_lambda() to create a custom callback to run at the beginning or end of batches or epochs, or based on whether you are training, predicting, or testing.
   It probably makes most sense to compute AUC at the end of each epoch, such as:

library(keras)

callback_auc <- function() {
  callback_lambda(
    on_epoch_end = function(epoch, logs) {
      # insert code for computing (and possibly saving) AUC here...
    }
  )
}

# When fitting the keras model
fit(
  x = x,
  y = y,
  callbacks = list(callback_auc)
)

Here is one example of a custom AUC callback.

[mattwarkentin]

Quick follow-up, I did a little more digging and I think this will work for computing AUC for PR curve:

library(keras)
library(tensorflow)
library(tidyverse)

y <- mtcars$vs %>% to_categorical()
x <- mtcars %>% select(-vs) %>% as.matrix()

model <- keras_model_sequential()

model %>% 
  layer_dense(10, input_shape = c(10)) %>% 
  layer_dense(units = 2, activation = "softmax")

summary(model)

model %>% 
  compile(
    optimizer = "rmsprop",
    loss = "categorical_crossentropy",
    metrics = tf$keras$metrics$AUC()
  )

model %>% 
  fit(
    x = x,
    y = y,
    epoch = 100
  )