Okay that is what I expect. Now, let's add 0.5 to each x-axis value so we have a non-integer scale (note I used as.numeric() above to the x-axis is not an integer type.)
Are you running ggplot 3.3.0? If so, ggplot now tries to determine the orientation from the aesthetic mapping (see "Bi-directional geoms and stats" here) and it looks like it is guessing wrong (or at least differently than you intended) here.