geom_col() doesn't perform any count or aggregation in your data, that is why you have to provide a y aesthetic with absolute values, your code is essentially stacking a little column (and text label) for each row in the dataset,if you want to perform summaries you have to operate manually in the dataset, see this example
library(tidyverse)
iris %>%
group_by(Species) %>%
summarise(mean_sepal_width = mean(Sepal.Width)) %>%
ggplot(aes(x = "1", mean_sepal_width, fill = reorder(Species, mean_sepal_width), label = mean_sepal_width)) +
geom_col() +
geom_text(size = 3, position = position_stack(vjust = 0.5))

Created on 2019-08-11 by the reprex package (v0.3.0.9000)