Thanks Hadley,
I intutively mapped geom_text() into a mental category of 'texty' things which have font sizes, not into the category of 'data-pointy' things which don't. Depending on what proportion of people put this in what mental category it might make more sense from a user-friendliness perspective to default to font sizes and have people transform it by the magic number to get mm if they want.
I agree with the poster of that issue that expanding themes to affect non-data components seems like a good idea.