Within-class accuracy metrics for multi-class models

I have a question regarding accuracy metrics for a multi-class model. Unless I'm missing something it seems like there isn't any support in yardstick for within-class accuracy metrics for a multi-class model.

Here's some data to motivate this:


counts <- tribble(
    ~truth, ~pred, ~n,
    "A", "A", 20,
    "A", "B", 3,
    "A", "C", 0,
    "B", "A", 4,
    "B", "B", 23,
    "B", "C", 6,
    "C", "A", 8,
    "C", "B", 5,
    "C", "C", 29

pred_df <-
    counts %>%
    uncount(n) %>%

Here's the confusion matrix for that data:

(yes, I realize that yardstick::conf_mat() exists :slight_smile:)

cm <- with(pred_df, table(Prediction = pred, Truth = truth))

#>           Truth
#> Prediction  A  B  C
#>          A 20  4  8
#>          B  3 23  5
#>          C  0  6 29

We can obtain relevant micro/macro/macro weighted accuracy metrics with something like this:


my_metrics <- metric_set(precision, recall)

my_metrics(pred_df, truth = truth, estimate = pred, estimator = "macro")
#> # A tibble: 2 x 3
#>   .metric   .estimator .estimate
#>   <chr>     <chr>          <dbl>
#> 1 precision macro          0.732
#> 2 recall    macro          0.752

Now, if I want to look at precision and recall at the class level rather than aggregate then it seems like I need to step outside of the tidymodels workflow, which would be a bummer with how smooth and seamless the current workflow is, especially when incorporating resampling and gathering all of the metrics with tune::collect_metrics().

# precision
cm %>% prop.table(margin = 2) %>% diag()
#>         A         B         C 
#> 0.8695652 0.6969697 0.6904762

# recall
cm %>% prop.table(margin = 1) %>% diag()
#>         A         B         C 
#> 0.6250000 0.7419355 0.8285714

So, I guess my question is: does anyone know if within-class metrics are currently/will be supported? If not, does anyone have suggestions on how to incorporate them w/out having to break the tidymodels workflow?

P.S. Perhaps this post is better served as a feature request on GitHub, I'll add something there if so.

Yes. An issue would be better.

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.