A reprex to demonstrate my point. Setting geom_text(size = 12) results in a lot bigger of change than setting theme(axis.title = element_text(size = 12). It appears that the font size specified in theme() is close to what a 12pt font might resemble in a word editor. Is there a specific reason for this behavior?
A solid discussion on how size works in ggplot2 works probably require a blog post (which I'd love to write, but this month is a difficult time for me, sorry!). But my understanding is that the most important difference here is that the size parameter in geoms:
(a) works on the data scale, not the plot scale, and
(b) uses different units depending on the shake being plotted (points, text, polygons, etc).
When I say that geoms' size works on the data scale, what I mean is that the plotting process translates data values (Sepal.Length and Sepal.Width) to "size on paper" values. Does that make sense?
... . The unit of size in ggplot2 is mm, not pt, and I've just redefined .pt as 72 / 25.4 to hopefully make that more clear in the code – hadleyAug 6 '15 at 11:51
And I've now changed it to 72.27 / 25.4 since as pointed out in the comments, grid uses printers' pointers, not Adobe/big points. – hadleyAug 6 '15 at 11:56
ggplot2's magic number is point size / mm (i.e. 72.27 / 25.4)