A reproducible example would help - you'd likely discover that if your PATIENT SEX column is numeric then your standalone boxplot doesn't have groups (especially not two labelled "0" and "1"). Converting the SEX variable to factor is the first step towards getting this right. Here's a reprex which shows it working. Hope that helps.
library(ggplot2)
library(ggsignif)
set.seed(1)
d <- data.frame(SEX = c(rep(0, 10), rep(1, 10)),
AGE = c(sample(20:40, 10, replace = TRUE), sample(30:50, 10, replace = TRUE)))
ggplot(d, aes(factor(SEX), AGE)) +
geom_boxplot() +
geom_signif(comparisons = list(c("0", "1")),
map_signif_level=TRUE)
#> Warning in wilcox.test.default(c(23, 26, 20, 21, 30, 33, 37, 38, 20, 40), :
#> cannot compute exact p-value with ties
Created on 2020-06-16 by the reprex package (v0.3.0)