This is special and specific layout that you need. flexdashboard feature are levering a special markdown syntax to create a navbar layout, by using the h1 level (#) as navbar component.
tabset component as you used is a special feature for html_document also leveraging some markdown syntax.
But the two are not compatible without respecting flexdashboard special markup. Tabset are supported with Flexdashboard but following this way https://pkgs.rstudio.com/flexdashboard/articles/using.html#tabsets
We don't have a format that is in between (have navbar created with # header without other flexdashboard feature).
However, you can have navbar layout using website structure. Did you try already R Markdown website
It is more oriented toward website project but it can help creating a navbar. You have to split your one Rmd in several though, or use links to section in the navbar. it may require tweaks to get exactly what you need.
You should be able to use navbar with a single document too if you provide a _navbar.html with HTML code directly, or _navbar.yml with YAML definition has in website doc. This will get picked up by html_document. However, if will still require some tweaks if you want to have h1 be transformed in navbar component.
More complex though and not as straightforward