Adjusting posterior model estimated probabilities after re-balancing or applying case weight

[Max]

@konradino

Of course the recall-precision trade-off was much better in the up-sampled model because of a higher recall and that completely makes sense when the cost of approving bad credit is so much higher than rejection a good one, but I was very surprised to see so little effect on the AUC - the difference was almost negligible.

That has been my experience with any type of subsampling for class imbalances; the default cutoff is more reasonable and the AUC is often equivalent to the original model.

@konradino

Does that eventually mean that in credit/ fraud scoring applications up-sampling would not necessarily be so required because the relative ordering of good/ bad predictions is preserved even without up-sampling as the AUC suggestes?

No, I wouldn't say that. There is an effect of the sampling methods of the model; it isn't just the same model with different weights/coefficients (see footnote).

There is a positive effect on the calibration of the model in the sense that the classification (i.e. posterior) probabilities do not have such pathological distributions.
I tried to demonstrate this on slide 14 of the classification notes found here.

@konradino

That also means that any future estimate of the model would not reflect the true, imbalance nature of the original training set, correct? In this case, is there a 'statistically' valid way of adjusting estimated posterior probabilites of future predictions?

Yes to the first question (and yes is a good thing). Without rebalancing, it wouldn't be uncommon to have your mostly likely event to have a class probability less than 60% (or lower).

I wouldn't readjust the class probabilities further (since that is what the subsampling is accomplishing). If you don't subsample, you could use another recalibration method (using Bayes theorem or monotonic regression); I tend to use subsampling to solve this issue though.

[footnote] The exception to this is ordinary logistic regression. The class imbalance drives the intercept to an extreme and rebalancing the data only effects the intercept parameter (and all of the standard errors).