Change the font in the theme.
library(ggplot2)
ggplot(mtcars, aes(x=wt, y=mpg)) + geom_point() +
ggtitle("Fuel Efficiency of 32 Cars") +
xlab("Weight (x1000 lb)") + ylab("Miles per Gallon") +
theme(text=element_text(size=16, family="gentium"))
I used **Gentium** as I am on Linux and do not have Calibri or Times New Roman installed.
Example taken from Change fonts in ggplot2, and create xkcd style graphs | R-bloggers