A solid discussion on how size works in ggplot2 works probably require a blog post (which I'd love to write, but this month is a difficult time for me, sorry!). But my understanding is that the most important difference here is that the size parameter in geoms:
(a) works on the data scale, not the plot scale, and
(b) uses different units depending on the shake being plotted (points, text, polygons, etc).
When I say that geoms' size works on the data scale, what I mean is that the plotting process translates data values (Sepal.Length and Sepal.Width) to "size on paper" values. Does that make sense?
(I'm by no means an expert here, mind!)