How not to capture click when brushing? Ideally how to emulate an 'on release' to capture click events...?

hello,

I have my user annotate a ggplot graph, think not point selection but more 'picture labelling'.
Labelling is either a click or a drag action to draw a rect.
Ideally, when the user wants to label an aera, I'd love not to have the user have to check an option before doing so. Which is what i'm considering because I can't find a way to not trigger click when I'm brushing...

For example, server side, I can't have a condition like

if (!is.null(last_click()) && is.null(last_drag()) {
_put points only when no dragging is going on_
}

Groups discussion on the subject, github feature request discussion...

As click labelling is my main usage (90% of the time), forcing double click for that is not an option, so my escape is with a checkBox to switch between click and drag...
Thx for the help.

(Yes, I've also tried the idea to remove the undesired click once the brush is released, but it's not super clean visually...)