I stumbled on the answer to my own problem, so I'll mark this closed. The example I needed was in Hadley's Shiny book, which I was looking at for a solution to a totally different problem.
He had an introduction to each part, which was its own .Rmd file. When i did it that way, it seemed to work.
As an aside, I also discovered that if you're trying to control the naming of the html files, you can't start them with numbers. Or at least it seems that way.
It looks like this:
# (PART\*) Getting started {.unnumbered}
# In this section... {#starter-intro .unnumbered}
Thanks again for considering helping me. I'll mark this ask closed