About this, as I think it is the issue you meant initially that I can reproduce, this is because rmarkdown::render() with no output_format passed will use the first one found in _output.yaml + YAML header in the file. I bet you wanted the otherway around but it is like that that R Markdown works from a long time.
If you call rmarkdown::render("README.Rmd", "github_document") it will correctly render using the format github_document using the option you may have defined inside the document YAML header itself.
So for a bookdown project, I think the best solution would be to call this when you want to render your document using the other format.
This is an issue (or feature) from rmarkdown and it is not speific to bookdown it seems. I think it will happens with other type of project with similar structure.
If not already filled as a Github issue somewhere, maybe it is worth opening;