By default, reprex renders as markdown (which is why it shows up nicely formatted here, on SO, GitHub, etc.).
There is reprex(..., venue = "html"), but that wouldn't modify the YAML exactly as you're describing. I would, however, recommend giving that a whirl before digging in more deeply.
More details on venues, etc. in the README here:
The RMarkdown templates used in reprex are here.