Plot point at median of boxplot with color mapping and varwidth = TRUE

When trying to plot a point at the medians of boxplots separated by x and color mapping, I've been unable to get them align on the x axis when varwidth = TRUE. A quick example:

library(ggplot2)
packageVersion("ggplot2") # Need v 2.2.1.9000, available on GitHub
library(dplyr)
iris2 <- mutate(iris, sw = Sepal.Width < 3.2, Species.sw = paste(Species, sw, sep = "."))

g <- ggplot(data = iris2, aes(Species, Sepal.Length, color = sw)) +
  geom_boxplot(varwidth = TRUE) 
g + stat_summary(fun.y = median, geom = "point", stroke = 2.25, shape = 18)

My closest approximation is to combine the two factor variables and plot that on the x-axis. This works for my purposes, but it's not quite as clean as I'd like. Is there a way to do this without this workaround?

ggplot(iris2, aes(Species.sw, Sepal.Length, color = sw)) +
  geom_boxplot(varwidth = TRUE) +
  stat_summary(fun.y = median, geom = "point", stroke = 2.25, shape = 18) 

(I'm a new user and can only post one image per post. Link to plot.)

I've already tried using the position argument for stat_summary, but it won't line up when the widths of the boxplots aren't the same

g + stat_summary(fun.y = median, geom = "point", stroke = 2.25, shape = 18, position = position_dodge2(width = 0.35))

Plot