As I understand the situation, the geoms generally have an argument named inherit.aes with a default value of TRUE. Any settings in the default aes set in ggplot() will be shared in the geom. If you set inherit.aes to FALSE, the geom's aes will be entirely independent.
geom_point(
mapping = NULL,
data = NULL,
stat = "identity",
position = "identity",
...,
na.rm = FALSE,
show.legend = NA,
inherit.aes = TRUE
)