Thanks. I found a way to move the segment text, but I'm still interested in converting my ggplots to grobs for advanced editing purpose. I think @DavoWW 's mtcars example is a good one. To make a very simple change, here's how I can change the background color of the plot.
But beyond this, I'm at a loss as to how I would search for a specific element and edit it.
library(ggplot2)
library(grid)
p1 <- ggplot(mtcars, aes(x=disp, y=mpg)) + geom_point()
p1
is.grob(p1) # Returns false
p2 <- ggplotGrob(p1)
p2$grobs[[1]] <- editGrob(p2$grobs[[1]], gp = gpar(fill = "blue"))
grid.draw(p2)

For example, perhaps searching for the bottom axis ticks to move them up above the axis line. But any example will do.