AROC for classes in Multi-Class models

[jka_ibx]

Following the example

library(yardstick)
library(dplyr)
library(ggplot2)

hpc_cv %>%
  filter(Resample == "Fold01") %>%
  roc_curve(obs, VF:L) %>%
  autoplot()

Displays the roc curves, however, I would also like the individual aucs for the accompanying diagram, however, if I pass like this:

hpc_cv %>%
  filter(Resample == "Fold01") %>%
  roc_auc(obs, VF:L)

I get some type of aggregate as opposed to individual rocs - do I need to write a function to extract these or is there a different function to utilize ?

[mattwarkentin]

Hi @jka_ibx,

This is the nature of how those functions work. roc_curve() provides a curve for each class, while roc_auc() will provide an aggregate statistic using, by default, the Hand-Till method. This can be controlled by the estimator argument.

If you want to compute an AUC per class (using the one vs. everything approach), you can manually integrate the step functions from roc_curve() or do some data restructuring to use roc_auc() (or roc_auc_vec()).

library(yardstick)
library(tidyverse)

hpc_cv %>%
  filter(Resample == "Fold01") %>%
  select(obs, VF:L) %>% 
  pivot_longer(-obs, names_to = 'class', values_to = 'prob') %>% 
  group_nest(class) %>% 
  mutate(
    auc = map2_dbl(class, data, function(x, y) {
      obs <- factor(y$obs == x)
      est <- y$prob
      roc_auc_vec(obs, est, event_level = 'second')
    })
  )
#> # A tibble: 4 × 3
#>   class               data   auc
#>   <chr> <list<tibble[,2]>> <dbl>
#> 1 F              [347 × 2] 0.817
#> 2 L              [347 × 2] 0.929
#> 3 M              [347 × 2] 0.812
#> 4 VF             [347 × 2] 0.928