I think 0.05 to 0.005 to 0.0005 are too small to see any change.
If that ggplot2 Quick Reference: size link above is approximately correct, "A size of 1 corresponds to approximately 0.75 mm".
And so size = 0.05 is 0.0375mm.
On an iPhone or retina mac, a pixel is 0.09652mm. So your line thickness is less than half a retina pixel.
As with so many things in life, I think you're just going to want to play with it until you get 'er looking right.
Here's a few options I messed with:
#setup
library(maps);library(dplyr);library(ggplot2)
usa <- map_data("usa")
#too small
usa %>% ggplot() +
geom_polygon(
aes(x=long, y = lat, group = group), colour = "red",
size = 0.001)

#too big
usa %>% ggplot() + geom_polygon(
aes(x=long, y = lat, group = group), colour = "red",
size = 1)

#nice looking...?
usa %>% ggplot() + geom_polygon(
aes(x=long, y = lat, group = group), colour = "red",
size = 0.15)
