How can I add PRAUC as a custom summaryFunction to Caret?

[dougfir]

I'm reading the Caret documentation here about adding alternate performance metrics.

    my_summary  <- function(data, lev = NULL, model = NULL){
      a1 <- defaultSummary(data, lev, model)
      b1 <- twoClassSummary(data, lev, model)
      c1 <- prSummary(data, lev, model)
      d1 <- MLmetrics::PRAUC(y_pred = data[, "pred"], y_true = data[, "obs"])
      out <- c(a1, b1, c1, d1)
      out}

Then in trainControl:

    train_control <- trainControl(
      method = "cv",
      number = 5,
      classProbs = TRUE,
      verboseIter = TRUE,
      savePredictions = TRUE,
      allowParallel = TRUE
    )

If I exclude d1, everything works great. However, when I try to use MLmetrics::PRAUC() I get an error "Error in { : task 1 failed - "Format of predictions is invalid."

How can I add PR AUC as a custom function to Caret, either using MLmetrics or not?

Rest of XGB via Caret code:

    ## tuning grid
    tune_grid <- expand.grid(nrounds = 200,
                             max_depth = 5,
                             eta = 0.05,
                             gamma = 0.01,
                             colsample_bytree = 0.75,
                             min_child_weight = 0,
                             subsample = 0.5)
    
    ## xgb
    ### pre process does not work with formula interface, only x, y
    XGBs <- foreach(i = targets) %do% {
      target <- as.factor(median_training_data[i][[1]]) %>% make.names()
      xgb <- train(x = select(median_training_data, -i), 
                   y = target,
                   method = "xgbTree",
                   trControl = train_control,
                   tuneGrid = tune_grid,
                   tuneLength = 10)
      saveRDS(xgb, file = paste0(i, ".rds"))
      return(1) # make loop not store in memory
    }

[Max]

You can use the option summaryFunction = prSummary in trainControl to get this.

[dougfir]

This is great, thank you very much!