The idea of knitting .Rmd once, keeping the intermediate .md, and converting it three times via Pandoc is certainly sounds much more efficient than compiling Rmd three times. Unfortunately, there is a fundamental problem. By default, rmarkdown::render() uses different figure and cache directories for different output formats. For example, the figure format for PDF outputs is pdf, and figures are generated to foo_files/figure-latex/; for HTML output, the figure directory is foo_files/figure-html.
That means for different output formats, the intermediate .md file will be different, unless you don't have figure output or forces all output formats to use the same figure format, in which case you could use clean = FALSE and run_pandoc = FALSE as @cderv mentioned, and then call rmarkdown::pandoc_convert() separately.
It really depends on how time-consuming your code chunks are. Personally I think the easiest solution is to turn on caching on these chunks, and cache them three times for the three output formats, respectively. The next time it should be fairly fast to regenerate the output files if you don't modify these code chunks.