Hi there,
A reprex or even just a copy/paste of your code would be easier to work off of, for future reference.
In any case, I think the code below is roughly what you're looking for with particular regards to the inherit.aes argument in geom_smooth in addition to the other arguments you might specify about the method, standard error, etc.
ggplot(your_data, aes(var1, var2, var3 ...)) +
geom_point() +
geom_smooth(data=your_data, aes(var1, var2), inherit.aes=F, ...)
The details: geom_*() functions automatically inherit aesthetics defined in ggplot(data, aes()). This is generally the preferred approach to using ggplot(), but you can write out aesthetics individually in each geom_*() function or use inherit.aes=F to circumvent this standard approach.