Hi, ggplot's geom_smooth() does the fitting for you, no need to fit a linear model first nor to specify a formula explictly in your case. Alternatively, you can add the line manually but not with geom_smooth() but geom_line().
library(tidyverse)
y1=c(10.05,1.5,-1.234,0.02,8.03)
x1=c(-3,-1,0,1,3)
data1<-data.frame(y1,x1)
ggplot(data1, aes(x = x1, y = y1)) +
geom_point(size = 3, shape = 19, color = "red") +
geom_smooth(method = "lm", se = FALSE, linetype = "solid", color = "blue", size = 3)
#> `geom_smooth()` using formula 'y ~ x'

Fit1<-lm(y1~x1,data = data1)
yhat1<-predict(Fit1,data1)
yhat2 <- as_tibble(x = x1, y = yhat1)
ggplot(data1, aes(x = x1, y = y1)) +
geom_point(size = 3, shape = 19, color = "red") +
geom_line(data = yhat2, aes(x = x1, y = yhat1), color = "blue", size = 3)

Created on 2020-07-16 by the reprex package (v0.3.0)