This happens because ggplot is automatically resizing the plot area to accommodate the legend. The overall width of the plot objects is the same, but the legends are different widths, so the areas of the plots within those windows get resized and end up differing.
For some possible solutions, check out this answer on Stack Overflow.