Did you do any manual resizing of the image after dropping it into Word? This will affect the apparent font size. I don't think you can embed PDF or SVG images into Word. They get "flattened" into PNGs which means you can't modify the figure text.
You need to set the font size in ggplot(...) and set the image height and width to whatever size you want the figure within the Word doc (figure this out before saving) when exporting, using ggsave(...).
For example, if you want the figure to be 4in x 3in in the Word doc:
p <-
ggplot(iris, aes(Sepal.Length)) +
geom_histogram() +
theme(text = element_text(size = 11))
ggsave('image.png', p, width = 4, height = 3, units = "in")